Model prefix parameter and hyper-parameter joint optimization method and system
By searching for the optimal prefix parameters in a low-dimensional space and dynamically adjusting the hyperparameters, the problems of low optimization efficiency and improper hyperparameter settings in high-dimensional parameter spaces are solved, thereby improving the model accuracy and stability in medical question answering tasks.
Patent Information
- Application Number
- CN202511721098.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-03-03
AI Technical Summary
In medical question-answering tasks, existing technologies struggle to efficiently optimize model prefix parameters and hyperparameters, resulting in high fine-tuning costs, slow convergence, and low efficiency in setting hyperparameters. It is difficult to find the optimal configuration with limited computing resources, especially in vertical fields with high accuracy requirements, where there is a lack of intelligent hyperparameter optimization mechanisms.
We use a random projection matrix to compress high-dimensional prefix parameters into a low-dimensional latent space. We combine a covariance matrix adaptive evolution strategy and a particle swarm optimization algorithm to search for prefix parameters, dynamically adjust the learning rate and weight decay, and optimize hyperparameters through a cosine annealing strategy to achieve a closed-loop process of "compression-optimization-reconstruction".
It significantly reduces the dimensionality of prefix parameter optimization, improves the accuracy and stability of the model in medical question answering tasks, solves the problem of manual hyperparameter tuning in traditional methods, and improves the efficiency and generalization ability of prefix tuning.
Smart Images
Figure CN121599050A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, specifically to a method and system for joint optimization of model prefix parameters and hyperparameters. Background Technology
[0002] The widespread application of large-scale language models in various natural language processing tasks has led to a continuous expansion of their parameter size, resulting in a significant increase in fine-tuning costs. In the medical field, how to efficiently inject vertical domain knowledge while maintaining the original capabilities of the model in medical question-answering tasks has become one of the current research hotspots.
[0003] To address this, efficient parameter fine-tuning methods have emerged, with prefix tuning attracting particular attention due to its ability to optimize a set of continuously trainable prefix vectors without modifying the main model parameters, requiring only training a small number of prefix vectors, and freezing the remaining model parameters. Traditional prefix tuning methods typically optimize a set of continuous trainable prefix vectors to achieve optimal performance on a specific task. However, this direct optimization in a high-dimensional parameter space suffers from problems such as high dimensionality of the prefix parameter space, a large number of trainable parameters, and slow convergence. Furthermore, traditional methods often rely on gradient descent algorithms for training, requiring manual tuning of key hyperparameters such as learning rate and weight decay. Different hyperparameter combinations significantly impact model performance, and traditional grid search or random search methods are inefficient and struggle to find the optimal configuration with limited computational resources. Especially in scenarios with extremely high accuracy requirements, such as vertical domains, further improving the optimization efficiency and generalization ability of prefix parameters and solving the hyperparameter optimization problem has become a pressing technical challenge.
[0004] Building upon the above, researchers attempted to introduce evolutionary algorithms for black-box optimization of prefix parameters to avoid the risk of local optima caused by gradient descent. However, the search difficulty of evolutionary algorithms increases in high-dimensional parameter spaces, making it difficult to quickly find the optimal prefix parameters. Therefore, how to efficiently search for the optimal prefix parameters in a low-dimensional latent space has become a key technical path to improve the performance of prefix tuning. Meanwhile, existing research lacks effective methods for dynamic optimization of hyperparameters during training. Traditional learning rate scheduling strategies (such as cosine annealing and linear decay) can improve training results to some extent, but they cannot adaptively adjust based on the actual performance of the model on the validation set, lacking an intelligent hyperparameter optimization mechanism. Summary of the Invention
[0005] This invention aims to provide a method and system for joint optimization of model prefix parameters and hyperparameters, which can improve the accuracy and stability of prefix tuning in medical question answering tasks.
[0006] To address the aforementioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for joint optimization of model prefix parameters and hyperparameters, comprising the following steps: S1: Obtain the joint optimization instructions for high-dimensional prefix parameters and hyperparameters; S2: Generate a random projection matrix based on the joint optimization instruction of high-dimensional prefix parameters and hyperparameters, and map the high-dimensional prefix parameters to low-dimensional prefix parameters through the projection matrix; S3: Search for the optimal low-dimensional prefix parameters in the low-dimensional space through the covariance matrix adaptive evolution strategy, reconstruct the low-dimensional prefix parameters into dynamic high-dimensional MLP parameters, generate the key-value prefix embeddings required by each layer of the model and inject them into the large model. S4: Dynamically adjust training hyperparameters based on particle swarm optimization and cosine annealing strategy, and gradually transition hyperparameters during training; S5: Train the model based on the optimal prefix parameters and hyperparameters, and output the trained model; S6: Apply the trained model to a medical question-answering task.
[0007] By employing the aforementioned technical solution, a differentiable random projection matrix is constructed to compress high-dimensional prefix parameters into a low-dimensional latent space, within which CMA-ES is used for efficient global prefix parameter search. Secondly, a particle swarm optimization algorithm combined with a cosine annealing strategy is introduced to achieve dynamic adaptive optimization of key hyperparameters such as learning rate and weight decay during training. The optimized low-dimensional parameters are then reconstructed into high-dimensional prefixes through the projection matrix, achieving a closed loop of "compression-optimization-reconstruction." The PSO algorithm searches for the optimal hyperparameter combination through swarm intelligence and works in conjunction with the cosine annealing scheduler to achieve smooth transition and dynamic adjustment of hyperparameters. This method not only significantly reduces the optimization dimensionality of prefix parameters but also retains the expressive power of the high-dimensional space. Simultaneously, it solves the problem of manual hyperparameter tuning in traditional methods, improving the accuracy and stability of prefix tuning in medical question-answering tasks.
[0008] Optionally, the elements of the random projection matrix in S2 follow a uniform distribution, and are arranged according to... Scaling, where This represents the low-dimensional optimization dimension of the target.
[0009] Optionally, S3 includes: S31: Initialize evolutionary parameters and generate a candidate individual population; S32: Calculate the average cross-entropy loss of the validation set and convert it into perplexity, then convert the perplexity into fitness value; S33: Update the covariance matrix and adaptively adjust the step size; S34: Reconstruct the low-dimensional prefix parameters into dynamic high-dimensional MLP parameters, parse the high-dimensional MLP parameters into weights and biases, and dynamically construct an MLP network structure containing an input layer, activation function, and output layer.
[0010] Optionally, S4 includes: S41: Perform PSO search and update hyperparameters at training step intervals, using validation set perplexity as fitness, within the constraints of learning rate multiplier and weight decay. S42: Calculate the cosine annealing factor, update the learner's learning rate and weight decay in conjunction with the hyperparameters, and apply numerical stability constraints on single-step learning rate changes and absolute boundaries.
[0011] Optionally, S4 further includes: S43: When the learning rate of PSO search changes by more than 0.1, the hyperparameters are smoothly transitioned through cubic spline interpolation.
[0012] Optionally, the step of parsing high-dimensional MLP parameters into weights and biases, and dynamically constructing an MLP network structure including an input layer, an activation function, and an output layer includes: First layer weight Before extracting high-dimensional MLP parameters Each element is reshaped into... The weight matrix, where, Indicates all OK, A set of real matrices of columns; First layer bias Extracting high-dimensional MLP parameters One element; Second layer weights :extract middle Each element is reshaped into... The weight matrix; Second layer :extract Remaining One element, namely: in, Indicates the width of the intermediate hidden layer. This indicates the hidden dimension of the model. Indicates the prefix feature dimension; In network assembly, according to linear layer 1 Tanh Linear layer 2 is constructed sequentially, that is: .
[0013] Optionally, S41 includes: Determine if the current training steps have reached the PSO running interval. If yes, execute B1; otherwise, jump to B2. B1: Calculate the current inertia weight and update the velocity and position of all particles, constraining particle coordinates within the search boundary; Speed and position updates: Inertia weight Linear decay: in, Indicates the current iteration number. Indicates the maximum number of iterations; Speed updates: Cognitive coefficient Social coefficient : in, This represents the velocity vector of the i-th particle in generation t. This represents the current position vector of the i-th particle in generation t. This represents the optimal value for an individual. Indicates global optimum; This indicates the generation of a random number; Position update, boundary constraints: in, Indicates the first Dimensional boundary.
[0014] B2: Calculate the cosine decay factor and update the learning rate and weight decay; The formula for calculating the cosine attenuation factor is as follows: Update learning rate : in, This represents the minimum learning rate. This represents the effective learning rate multiple. Indicates the basic learning rate. Indicate the cosine annealing factor; update the weight decay. : .
[0015] Secondly, the present invention provides a joint optimization system for model prefix parameters and hyperparameters, used to implement a joint optimization method for model prefix parameters and hyperparameters as described in the first aspect, comprising: a data and model input module, a random projection module, a prefix optimization module, a hyperparameter optimization module, and an output module.
[0016] Optionally, the random projection module includes a random projection unit and a dimension transformation unit; the prefix optimization module includes a parameter initialization unit, a fitness evaluation unit, a distribution update unit, and a dynamic MLP reconstruction unit; and the hyperparameter optimization module includes a PSO hyperparameter search unit, a cosine annealing scheduling unit, and a smooth transition unit.
[0017] In summary, the present invention has at least the following beneficial technical effects: This invention achieves efficient mapping between high-dimensional and low-dimensional spaces through a random projection module, performs efficient optimization in the low-dimensional space through parameter optimization, and realizes structured reconstruction of parameters through dynamic MLP construction. Finally, it generates prefix features adapted to the main model. It can automatically adjust and optimize the learning rate and weight decay during training, solving the problem of low efficiency in high-dimensional parameter optimization. At the same time, it solves the problem of fixed learning rate and weight decay during training by using the PSO algorithm, maintaining the expressive power of prefix features and enhancing the performance of the model in vertical domains. Thus, it improves the accuracy and stability of prefix tuning in medical question answering tasks. Attached Figure Description
[0018] Figure 1 This is a flowchart of a joint optimization method for model prefix parameters and hyperparameters in an embodiment of the present invention; Figure 2 A flowchart illustrating the low-dimensional compression and reconstruction of high-dimensional MLP parameters for a random projection matrix.
[0019] Figure 3 Flowchart for CMA-ES adaptive evolutionary optimization; Figure 4 Flowchart for PSO-Cosine adaptive scheduling optimization; Figure 5 This is a schematic block diagram of a joint optimization system for model prefix parameters and hyperparameters in an embodiment of the present invention.
[0020] Figure labeling: 110, Data and model input module; 120, Random projection module; 121, Random projection unit; 122, Dimension transformation unit; 130, Prefix optimization module; 131, Parameter initialization unit; 132, Fitness evaluation unit; 133, Distribution update unit; 134, Dynamic MLP reconstruction unit; 140, Hyperparameter optimization module; 141, PSO hyperparameter search unit; 142, Cosine annealing scheduling unit; 143, Smooth transition unit; 150, Output module. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0022] In the description of this invention, it should be understood that the terms "high-dimensional prefix parameters", "low-dimensional prefix parameters", "projection matrix", "prefix embedding", etc. are all based on the functional definition of the invention. "High-dimensional prefix parameters" refers to the set of high-dimensional prefix parameters that need to be optimized in the original model, "low-dimensional prefix parameters" refers to the prefix parameters in the low-dimensional space obtained by projection compression, "projection matrix" refers to the transformation matrix that realizes the mapping between high-dimensional and low-dimensional spaces, and "prefix embedding" refers to the structured input features used to enhance model performance.
[0023] In the description of this invention, unless otherwise specified and limited, the terms "projection," "reconstruction," and "optimization" should be interpreted broadly: "projection" refers to mapping high-dimensional parameters to a low-dimensional space through a linear transformation; "reconstruction" refers to restoring low-dimensional prefix parameters to a high-dimensional space through an inverse transformation; and "optimization" refers to minimizing the target loss function by iteratively adjusting parameters. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.
[0024] This invention provides a method for joint optimization of model prefix parameters and hyperparameters, specifically designed for prefix parameter optimization of large pre-trained models (such as Transformer-type models). It can adaptively select the optimal prefix parameters, learning rate, and weight decay parameters for different models, solving problems such as low model prefix training efficiency, slow convergence speed, and inability to dynamically adjust the learning rate and weight decay during optimization in high-dimensional parameter spaces of large language models. This method can improve the accuracy and stability of prefix tuning in medical question answering tasks.
[0025] refer to Figure 1-4 A method for joint optimization of model prefix parameters and hyperparameters includes the following steps: S1: Data and model input module 110 obtains the joint optimization instructions for high-dimensional prefix parameters and hyperparameters.
[0026] After receiving the joint optimization instructions for prefix parameters and hyperparameters, the data and model input module 110 parses the instruction parameters and extracts the high-dimensional prefix parameter dimensions preset by the CMA-ES (Covariance Matrix Adaptive Evolution Strategy) optimizer from the instruction parameters. Target low-dimensional optimization dimension Calculate the device type and data type, and verify the dimensionality of the high-dimensional prefix parameter. Low-dimensional optimization of the target Does it meet the requirements? And compression ratio The constraints of "".
[0027] S2: The random projection module 120 generates a random projection matrix based on the joint optimization instructions of high-dimensional prefix parameters and hyperparameters, and maps the high-dimensional prefix parameters to low-dimensional prefix parameters through the projection matrix.
[0028] Specifically, refer to Figure 2 S2 includes the following steps: S21: Random projection unit 121 generates a uniformly distributed and scaled random projection matrix based on the Johnson-Lindenstrauss lemma.
[0029] Random projection matrix Satisfying the Euclidean distance approximation, it is used to achieve lossless compression of high-dimensional prefix parameters. Random projection matrix. The elements in the array follow a uniform distribution. .Right now, .according to Scaling the projection matrix ensures that the second-order matrix of the projected vector remains unchanged, satisfying the distance preservation condition. The projection matrix is then stored in a cache to ensure that the same matrix is used in subsequent "compression-reconstruction" operations, avoiding mapping deviations.
[0030] S22: Dimension transformation unit 122 maps high-dimensional prefix parameters to low-dimensional prefix parameters via a projection matrix.
[0031] Dimension transformation unit 122 treats the high-dimensional prefix parameters used to generate prefix embeddings as a high-dimensional parameter vector. Mapped to low-dimensional prefix parameters via projection matrix .Right now, ,like For a matrix, first pass through Vectorization Calculate after vector input; if batch input exists Then, multiplying each row by the projection matrix yields... .
[0032] in, Representative matrix Prefix features; This represents the number of vectors in the batch.
[0033] Finally, the compressed low-dimensional prefix parameters are output and stored in the prefix optimizer parameter space, thus providing the basic parameters of the prefix optimization module 130 after dimensional compression.
[0034] S3: The prefix optimization module 130 searches for the optimal low-dimensional prefix parameters in the low-dimensional space through an adaptive evolution strategy of the covariance matrix, reconstructs the low-dimensional prefix parameters into dynamic high-dimensional MLP (multilayer perceptron) parameters, generates the key-value prefix embeddings required by each layer of the model, and injects them into the large model.
[0035] Specifically, refer to Figure 3 S3 includes the following steps: S31: Parameter initialization unit 131 initializes evolutionary parameters and generates a candidate individual population.
[0036] Evolutionary parameters include: seed word feature extraction, population size, number of individuals selected, initial step size, covariance matrix C, etc.
[0037] In this embodiment, in order to obtain an initial search point that better fits the model feature distribution, an MLP parameter fitting is first performed in the low-dimensional space obtained by random projection: Let the low-dimensional prefix parameter vector be... Its initial value is randomly initialized, and then projected through a random projection matrix. Reconstructed into high-dimensional MLP parameters (including weights and biases of the first and second layers), using the target prefix features obtained by concatenating seed word key-value pairs. To monitor the signal, the Adam optimizer is used to minimize the MLP output and The reconstruction error between them yields the converged low-dimensional prefix parameter vector. After optimization, the low-dimensional prefix parameter vector is... As the initial mean vector of CMA-ES Based on this, a covariance matrix adaptive evolution strategy is further adopted for global search.
[0038] Seed word feature extraction involves selecting task-related seed words (such as "medical" and "diagnosis" in the medical field), inputting them into the pre-trained model, and extracting key-value pair features from the shallow attention mechanism. , in, For prefix length, This represents the dimension of the key vector for the attention mechanism. This represents the dimension of the attention value vector. .
[0039] Concatenating key-value pair features yields the target prefix feature: Vectorization .
[0040] Position encoding initialization, position encoding matrix: , in, The dimension of the position encoding vector. This represents the hidden dimension of the pre-trained language model.
[0041] In this example, It can be randomly initialized with trainable parameters, or the positional encoding vectors corresponding to the seed word sequence in the pre-trained model can be extracted and used as... The initial value. Position encoding vector. The calculation is as follows: in, Indicates position The unique heat vector.
[0042] Parameter initialization, initial mean: Provided by feature alignment initialization; Initial step size: It adaptively adjusts according to the parameter range; Initial covariance matrix: The identity matrix ensures isotropic initial search. Population size: To ensure search diversity; Weight: Elite individual weight, satisfying: in, For the number of elite individuals, effectively select quality .
[0043] In population generation and constraints, each generation generates... One candidate parameter: Apply range constraints to the candidate parameters: To avoid numerical overflow, adjustments can be made based on the model type.
[0044] S32: Fitness evaluation unit 132 calculates the average cross-entropy loss of the validation set and converts it into perplexity, and then converts the perplexity into fitness value to measure the quality of the parameters.
[0045] In fitness evaluation, the perplexity of the model on the validation dataset is used as the fitness metric; the lower the perplexity, the higher the fitness.
[0046] Conversion between perplexity and fitness: perplexity To maximize the fitness function, fitness is defined. To avoid exponential overflow, cross-entropy loss is used to calculate the average loss of the model on the evaluation set. The calculation formula is as follows: in, The number of valid tokens (excluding padding tokens with a value of -100). For input text, For tags.
[0047] S33: The distributed update unit 133 updates the covariance matrix and adaptively adjusts the step size to optimize the subsequent search direction to approximate the optimal prefix parameters.
[0048] The distributed update unit 133 performs CMA-ES optimization, which is divided into three stages: exploration stage, development stage, and refinement stage, to search for the optimal prefix parameters.
[0049] The adaptive phase switching proceeds from the exploration phase to the development phase and then to the refinement phase. In each generation, population generation, fitness assessment, distribution parameter updates, and convergence judgment are performed. The control optimization phase automatically adjusts the step size and population size according to the generation to improve search efficiency and accuracy.
[0050] When the distribution update unit 133 uses the covariance matrix adaptive strategy to update the distribution parameters, it selects the optimal individual and updates the mean vector, step size and covariance matrix C.
[0051] Mean update: , For the sorted elite individuals, As weight; Step size update, adaptively adjusts the search step size: in, The learning rate hyperparameter represents the step size update; the larger it is, the faster it changes. The damping coefficient represents the step size update; the larger it is, the slower the change. This represents the evolutionary path vector of generation t, which accumulates the search directions of the most recent generations. This represents a matrix with a mean of 0 and a covariance equal to the identity matrix. The standard multidimensional Gaussian distribution.
[0052] Covariance matrix update: in, This is the learning rate.
[0053] Then, the distribution update unit 133 determines whether the number of generations has reached the upper limit. If so, it executes step A1; otherwise, it executes step A2.
[0054] A1: Update the eigenvectors B and eigenvalues D of the covariance matrix through eigenvalue decomposition, and calculate the final optimal prefix parameters and fitness values.
[0055] The eigenvalue decomposition is expressed as: Where B is the eigenvector matrix (orthogonal matrix), and D is a diagonal matrix composed of the square roots of the eigenvalues. It is an eigenvalue diagonal matrix.
[0056] Eigenvalue decomposition is expressed as: , .
[0057] A2: Distributed update unit 133 determines whether the step size is less than the convergence tolerance or whether the fitness has reached the target. If so, execute S34; otherwise, execute A1.
[0058] S34: The dynamic MLP reconstruction unit 133 reconstructs the low-dimensional prefix parameters into dynamic high-dimensional MLP parameters, parses the high-dimensional MLP parameters into weights and biases, and dynamically constructs an MLP network structure containing an input layer, activation function, and output layer.
[0059] The dynamic MLP reconstruction unit 133 reconstructs low-dimensional prefix parameters into dynamic high-dimensional MLP parameters through a random projection matrix, and decomposes the high-dimensional MLP parameters into weights and biases of each layer of the MLP to construct a nonlinear mapping network, thereby realizing the transformation from low-dimensional prefix parameters to high-dimensional prefix features.
[0060] Specifically, a "two-layer linear transformation + Tanh activation" structure is adopted, with the input dimension being the hidden layer dimension of the model and the intermediate dimension... Output dimensions: in, For the number of model layers, The attention head key-value pair dimension.
[0061] High-dimensional MLP parameter parsing: reconstructing high-dimensional MLP parameters Decomposed into weights and biases according to the following rules ( (where are the weights and biases of the first and second layers, respectively). First layer weight :extract forward Each element is reshaped into... The weight matrix, where, Indicates all OK, A set of real matrices of columns; First layer bias :extract middle One element; Second layer weights :extract middle Each element is reshaped into... The weight matrix; Second layer :extract Remaining One element, namely: in, Indicates the width of the intermediate hidden layer. This indicates the hidden dimension of the model. This indicates the prefix feature dimension.
[0062] In network assembly, according to "linear layer 1" Tanh Linear layer 2 is constructed sequentially, that is: Finally, the system generates prefix embeddings from the optimized optimal prefix parameters, provides structured input through positional encoding, transforms and generates key-value prefix representations for each layer, and reshapes them into a format that adapts to the attention mechanisms of each layer of the model.
[0063] After evolutionary algebras and convergence judgment, this stage outputs the optimized optimal prefix parameters and convergence information, generates prefix embeddings and adjusts model parameters synchronously, and completes the global search for the optimal prefix parameters in the low-dimensional space.
[0064] S4: Hyperparameter optimization module 140 dynamically adjusts training hyperparameters based on particle swarm optimization (PSO) and cosine annealing strategy, gradually transitioning hyperparameters during training.
[0065] Specifically, refer to Figure 4 S4 includes the following steps: S41: The PSO hyperparameter search unit 141 performs PSO search and updates hyperparameters at training step intervals, with the validation set perplexity as the fitness, within the constraints of learning rate multiplier and weight decay.
[0066] The PSO hyperparameter search unit 141 first initializes the particle swarm parameters, including the number of particles, search dimensions (learning rate and weight decay), inertia weights, and cognitive / social coefficients.
[0067] Particle initialization: Optimization dimension is 2, corresponding to This is a multiple of the learning rate. For weight decay, the boundary constraints are as follows: , ; Particle position initialization: , For the first Dimensional boundary, velocity initialization , This indicates that a random number will be generated.
[0068] Then determine whether the current training steps have reached the PSO running interval. If so, execute B1; otherwise, jump to B2.
[0069] B1: Calculate the current inertia weights and update the velocities and positions of all particles, constraining particle coordinates within the search boundary.
[0070] Speed and position updates: Inertia weight Linear decay: in, Indicates the current iteration number. Indicates the maximum number of iterations; Speed of updates, cognitive coefficient Social coefficient : in, This represents the velocity vector of the i-th particle in generation t. This represents the current position vector of the i-th particle in generation t. This represents the optimal value for an individual. Indicates global optimum; Position update, boundary constraints: .
[0071] B2: Calculate the cosine decay factor and update the learning rate and weight decay.
[0072] The formula for calculating the cosine attenuation factor is as follows: Update learning rate : in, This represents the minimum learning rate. This represents the effective learning rate multiple. Indicates the basic learning rate. Indicates the cosine annealing factor; Update weight decay : S42: The cosine annealing scheduling unit 142 calculates the cosine annealing factor, updates the learning rate and weight decay of the optimizer in combination with the hyperparameters, and applies numerical stability constraints on single-step learning rate changes and absolute boundaries.
[0073] The system evaluates the fitness of each particle, updates the individual optimum and the global optimum, and records the number of particles that have improved.
[0074] In fitness calculation, fitness is the model's loss on the validation set. The cosine annealing scheduling unit 142 incorporates the cosine annealing factor during calculation. in: Indicates the basic learning rate. , This represents the current number of training steps. This represents the total number of training steps.
[0075] The system checks the difference between the new optimal parameters and the current parameters. If the change is too large, a smooth transition sequence is set; otherwise, the current multiple is updated directly.
[0076] S43: When the learning rate of PSO search changes by more than 0.1, the smooth transition unit 143 achieves smooth hyperparameter transition through cubic spline interpolation to avoid parameter mutations affecting training.
[0077] The system applies numerical stability constraints to limit the range of learning rate changes, outputs the optimized learning rate and weight decay parameters, and stores them in the optimizer parameter set.
[0078] If PSO is updated For changes exceeding 0.1, cubic spline interpolation is used to achieve a smooth transition. in, It is the number of smooth transition steps. , For PSO new optimal .
[0079] Numerical stability constraint: the change in learning rate in a single step does not exceed 50%. ; Absolute boundary constraints: To avoid the learning rate being too small or too large.
[0080] In addition, safeguards are provided during the optimization process, including loss function design to prevent overfitting, employing a combination of "SmoothL1 loss + L2 regularization": in, Indicates predicted value Represents the true value. The regularization coefficient is . For low-dimensional prefix parameters, This is the position encoding matrix.
[0081] SmoothL1Loss is robust to outliers and is defined as follows: or To avoid gradient explosion, for the prefix parameters (low-dimensional prefix parameters) Location coding L2 norm clipping is performed on the gradient of ). in, Adjust according to the gradient range of the model.
[0082] Input noise enhancement improves generalization by adding noise to the position-encoded input that decays as training progresses: in, This represents the encoding tensor of the current position. For the current epoch, the noise intensity decreases linearly to avoid the model relying too much on features at fixed locations.
[0083] This stage, through smooth transition and numerical stability constraints, ultimately outputs the optimized learning rate and weight decay parameters and stores them in the optimizer parameter set, realizing adaptive adjustment of training hyperparameters and ensuring the stability and efficiency of model training.
[0084] S5: Output module 150 trains the model based on the optimal prefix parameters and hyperparameters, and outputs the trained model.
[0085] The output module 150 integrates the optimal prefix parameters generated by prefix optimization and the adaptive learning rate and weight decay hyperparameters determined by hyperparameter optimization, and applies them in conjunction to the pre-training process to complete model training, and finally outputs the trained model.
[0086] S6: Apply the trained model to a medical question-answering task.
[0087] To verify the performance of the above method, experiments were conducted on a server (configuration: H800A (80G VRAM) *2, model: Qwen2.5-1.5B). The dataset used in the experiments was from Huggingface. The analysis considered the perplexity of the model's output answers in the relevant professional domain and human evaluations; lower loss and lower perplexity indicated higher accuracy of the model's output.
[0088] This invention achieves efficient mapping between high-dimensional and low-dimensional spaces through a random projection module 120, efficiently optimizes parameters in the low-dimensional space, and structurally reconstructs parameters through dynamic MLP construction, ultimately generating prefix features adapted to the main model. It can automatically adjust and optimize the learning rate and weight decay during training. This solves the problem of low efficiency in high-dimensional parameter optimization. Furthermore, by leveraging the PSO algorithm, it addresses the issue of fixed learning rate and weight decay during training, maintaining the expressive power of the prefix features and enhancing the accuracy and stability of the model in medical question-answering tasks.
[0089] This invention also provides a joint optimization system for model prefix parameters and hyperparameters.
[0090] Reference 5, a joint optimization system for model prefix parameters and hyperparameters, includes: a data and model input module 110, a random projection module 120, a prefix optimization module 130, a hyperparameter optimization module 140, and an output module 150.
[0091] The random projection module 120 includes a random projection unit 121 and a dimension transformation unit 122. The prefix optimization module 130 includes a parameter initialization unit 131, a fitness evaluation unit 132, a distribution update unit 133, and a dynamic MLP reconstruction unit 134. The hyperparameter optimization module 140 includes a PSO hyperparameter search unit 141, a cosine annealing scheduling unit 142, and a smooth transition unit 143.
[0092] The various variations and specific examples of the methods provided in the above embodiments are also applicable to the model prefix parameter and hyperparameter joint optimization system of this embodiment. Through the foregoing detailed description of a model prefix parameter and hyperparameter joint optimization method, those skilled in the art can clearly understand the implementation method of the model prefix parameter and hyperparameter joint optimization system of this embodiment. For the sake of brevity, it will not be described in detail here.
[0093] The above description of the embodiments is only used to provide a detailed introduction to the technical solution of the present invention. However, the description of the above embodiments is only for the purpose of helping to understand the method and core idea of the present invention, and should not be construed as a limitation of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be covered within the protection scope of the present invention.
Claims
1. A method for joint optimization of model prefix parameters and hyperparameters, characterized in that, Includes the following steps: S1: Obtain the joint optimization instructions for high-dimensional prefix parameters and hyperparameters; S2: Generate a random projection matrix based on the joint optimization instruction of high-dimensional prefix parameters and hyperparameters, and map the high-dimensional prefix parameters to low-dimensional prefix parameters through the projection matrix; S3: Search for the optimal low-dimensional prefix parameters in the low-dimensional space through the covariance matrix adaptive evolution strategy, reconstruct the low-dimensional prefix parameters into dynamic high-dimensional MLP parameters, generate the key-value prefix embeddings required by each layer of the model and inject them into the large model. S4: Dynamically adjust training hyperparameters based on particle swarm optimization and cosine annealing strategy, and gradually transition hyperparameters during training; S5: Train the model based on the optimal prefix parameters and hyperparameters, and output the trained model; S6: Apply the trained model to a medical question-answering task.
2. The method for joint optimization of model prefix parameters and hyperparameters as described in claim 1, characterized in that, The elements of the random projection matrix in S2 follow a uniform distribution, and are arranged according to... Scaling, where This represents the low-dimensional optimization dimension of the target.
3. The method for joint optimization of model prefix parameters and hyperparameters as described in claim 1, characterized in that, S3 includes: S31: Initialize evolutionary parameters and generate a candidate individual population; S32: Calculate the average cross-entropy loss of the validation set and convert it into perplexity, then convert the perplexity into fitness value; S33: Update the covariance matrix and adaptively adjust the step size; S34: Reconstruct the low-dimensional prefix parameters into dynamic high-dimensional MLP parameters, parse the high-dimensional MLP parameters into weights and biases, and dynamically construct an MLP network structure containing an input layer, activation function, and output layer.
4. The method for joint optimization of model prefix parameters and hyperparameters as described in claim 3, characterized in that, S4 includes: S41: Perform PSO search and update hyperparameters at training step intervals, using validation set perplexity as fitness, within the constraints of learning rate multiplier and weight decay. S42: Calculate the cosine annealing factor, update the learner's learning rate and weight decay in conjunction with the hyperparameters, and apply numerical stability constraints on single-step learning rate changes and absolute boundaries.
5. The method for joint optimization of model prefix parameters and hyperparameters as described in claim 4, characterized in that, S4 further includes: S43: When the learning rate of PSO search changes by more than 0.1, the hyperparameters are smoothly transitioned through cubic spline interpolation.
6. The method for joint optimization of model prefix parameters and hyperparameters as described in claim 3, characterized in that, The process of parsing high-dimensional MLP parameters into weights and biases, and dynamically constructing an MLP network structure including an input layer, activation function, and output layer, includes: First layer weight Before extracting high-dimensional MLP parameters Each element is reshaped into... The weight matrix, where, Indicates all OK, A set of real matrices of columns; First layer bias Extracting high-dimensional MLP parameters One element; Second layer weights :extract middle Each element is reshaped into... The weight matrix; Second layer :extract Remaining One element, namely: in, Indicates the width of the intermediate hidden layer. This indicates the hidden dimension of the model. Indicates the prefix feature dimension; In network assembly, according to linear layer 1 Tanh Linear layer 2 is constructed sequentially, that is: 。 7. The method for joint optimization of model prefix parameters and hyperparameters as described in claim 4, characterized in that, S41 includes: Determine if the current training steps have reached the PSO running interval. If yes, execute B1; otherwise, jump to B2. B1: Calculate the current inertia weight and update the velocity and position of all particles, constraining particle coordinates within the search boundary; Speed and position updates: Inertia weight Linear decay: in, Indicates the current iteration number. Indicates the maximum number of iterations; Speed updates: Cognitive coefficient Social coefficient : in, This represents the velocity vector of the i-th particle in generation t. This represents the current position vector of the i-th particle in generation t. This represents the optimal value for an individual. Indicates global optimum; This indicates the generation of a random number; Position update, boundary constraints: in, Indicates the first Dimensional boundary; B2: Calculate the cosine decay factor and update the learning rate and weight decay; The formula for calculating the cosine attenuation factor is as follows: Update learning rate : in, This represents the minimum learning rate. This represents the effective learning rate multiple. Indicates the basic learning rate. Indicates the cosine annealing factor; Update weight decay : 。 8. A joint optimization system for model prefix parameters and hyperparameters, characterized in that, The method for jointly optimizing model prefix parameters and hyperparameters as described in any one of claims 1-7 includes: a data and model input module (110), a random projection module (120), a prefix optimization module (130), a hyperparameter optimization module (140), and an output module (150).
9. The method for joint optimization of model prefix parameters and hyperparameters as described in claim 8, characterized in that, The random projection module (120) includes a random projection unit (121) and a dimension transformation unit (122). The prefix optimization module (130) includes a parameter initialization unit (131), a fitness evaluation unit (132), a distribution update unit (133), and a dynamic MLP reconstruction unit (134). The hyperparameter optimization module (140) includes a PSO hyperparameter search unit (141), a cosine annealing scheduling unit (142), and a smooth transition unit (143).