Model fine-tuning method, device and electronic equipment
By introducing a hybrid expert architecture of multiple LoRA modules and gating modules into the target layer of the pre-trained model, the limitations of LoRA's expressive power in complex task scenarios are solved, achieving stronger adaptability and generalization ability, and reducing computational resource consumption.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- MICRO DREAM TECHTRONIC NETWORK TECH CHINACO
- Filing Date
- 2026-01-12
- Publication Date
- 2026-06-05
Smart Images
Figure CN122154826A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence, and in particular to a model fine-tuning method, apparatus, and electronic device. Background Technology
[0002] In the era of large models, full fine-tuning can achieve good performance, but due to the enormous number of parameters (usually reaching billions or even hundreds of billions), the computational cost and storage requirements of full fine-tuning are extremely high. Therefore, parameter-efficient fine-tuning (PEFT) techniques have emerged. These techniques adapt to specific tasks by adjusting only a small number of parameters, thereby significantly reducing computational resource consumption. Low-rank adaptation (LoRA) is one of the most representative and widely used methods among them.
[0003] While LoRA improves training efficiency through low-rank decomposition, and large models reduce training parameter overhead by training only low-rank matrices (A, B), it also reduces storage space overhead. However, because LoRA only fine-tunes a subset of parameters (i.e., the low-rank matrix), it cannot fully utilize the potential of all parameters in the pre-trained model and struggles to cover complex task spaces. Summary of the Invention
[0004] This application provides a model fine-tuning method, apparatus, and electronic device to address the problem that LoRA in related technologies struggles to cover complex task spaces.
[0005] In a first aspect, embodiments of this application provide a model fine-tuning method, including: A pre-trained model is obtained, and a parameter fine-tuning module is set in the target layer of the pre-trained model; wherein, the parameter fine-tuning module includes multiple LoRA modules and a gating module, and the gating module is used to select a preset number of LoRA modules, each of the LoRA modules serving as an expert; Based on the input of the target layer, the model preset configuration, and the parameters of the parameter fine-tuning module, the parameters of the target layer are fine-tuned to obtain the fine-tuned output of the target layer; Based on the fine-tuned output and the base output of the pre-trained model, a fine-tuned model is obtained, wherein the base output is the output corresponding to the frozen parameters in the pre-trained model.
[0006] Secondly, embodiments of this application provide a model fine-tuning device, including: An acquisition module is used to acquire a pre-trained model and set a parameter fine-tuning module in the target layer of the pre-trained model; wherein, the parameter fine-tuning module includes multiple LoRA modules and a gating module, and the gating module is used to select a preset number of LoRA modules, each of the LoRA modules serving as an expert; The fine-tuning module is used to fine-tune the parameters of the target layer based on the input of the target layer, the model preset configuration, and the parameters of the parameter fine-tuning module, so as to obtain the fine-tuning output of the target layer; The determination module is used to obtain a fine-tuned model based on the fine-tuned output and the base output of the pre-trained model, wherein the base output is the output corresponding to the frozen parameters in the pre-trained model.
[0007] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method described in the first aspect.
[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.
[0009] Fifthly, embodiments of this application provide a computer program product stored in a computer-readable storage medium, which is executed by at least one processor to implement the steps of the method described in the first aspect.
[0010] In this embodiment, a pre-trained model is first obtained, and a parameter fine-tuning module is set in the target layer of the pre-trained model. The parameter fine-tuning module includes multiple LoRA modules and a gating module. The gating module selects a preset number of LoRA modules, each acting as an expert. Then, based on the input of the target layer, the model's preset configuration, and the parameters of the parameter fine-tuning module, the parameters of the target layer are fine-tuned to obtain the fine-tuned output of the target layer. Finally, based on the fine-tuned output and the base output of the pre-trained model, a fine-tuned model is obtained. The base output corresponds to the output of the frozen parameters in the pre-trained model. This embodiment sets multiple LoRA modules and a gating module in the target layer of the pre-trained model, and uses the gating module to select a preset number of LoRA modules. This means the gating module can select a preset number of experts based on the input, enabling the model to overcome the limitations of a single LoRA module's expressive power. By flexibly scheduling different experts, the model can have stronger adaptability in complex task scenarios. Attached Figure Description
[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the LoRA calculation process provided in one embodiment of this application; Figure 2 This is a flowchart of a model fine-tuning method provided in one embodiment of this application; Figure 3 This is a schematic diagram of the model structure of a decoder class large model provided in one embodiment of this application; Figure 4 This is a schematic diagram of the structure of a self-attention layer provided in one embodiment of this application; Figure 5 This is a schematic diagram of the structure of a model fine-tuning method provided in one embodiment of this application; Figure 6 This is a schematic diagram of the model output result provided in one embodiment of this application; Figure 7 This is a schematic diagram of the structure of a model fine-tuning device provided in one embodiment of this application; Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation
[0012] The embodiments of this application will now be described in detail. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0013] The terms "first" and "second" in the specification and claims of this application may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise stated, "multiple" means two or more. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0014] While full-parameter fine-tuning is effective in related technologies, it suffers from the following problems: The number of parameters is enormous; for example, fine-tuning all parameters of a 7-byte language model requires a large amount of GPU memory. Storage costs are high, as a complete set of model parameters needs to be saved for each fine-tuning task. Inference deployment is complex, with different tasks corresponding to different model copies, making unified management difficult.
[0015] To address these issues, efficient parameter fine-tuning techniques have emerged. LoRA is one of the most representative and widely used methods among them.
[0016] LoRA replaces certain weight matrices in the model with a low-rank decomposition form. For the weight matrix W in the pre-trained model, it can be represented as: W=W0+ΔW Where W0 is the original pre-trained weight, W0∈R m×n During training, this parameter is frozen and does not participate in gradient updates. ΔW is the offset matrix learned during fine-tuning. Using low-rank decomposition, ΔW can be represented as the product of two low-rank matrices A and B: ΔW=A B, W0∈R m×n Where, ΔW∈R m×n , A∈R m×r , B∈R r×n , rank r min(m,n). That is, the ranks of A and B are very low, which can greatly reduce the number of parameters.
[0017] Therefore, the weight matrix W can be expressed as: W=W0+A B In this way, only low-rank matrices A and B need to be learned, which can significantly reduce the number of parameters and computational resources required during fine-tuning.
[0018] During model training, only the parameters of A and B need to be trained, while W0 remains unchanged. The calculation process of LoRA is as follows: Figure 1 As shown.
[0019] While LoRA improves training efficiency through low-rank decomposition, and large models reduce training parameter overhead by training only low-rank matrices (A, B), it also reduces storage space requirements. However, this method has several significant drawbacks: Limited expressive ability Because LoRA only fine-tunes a subset of parameters (i.e., the low-rank matrix), it may not fully utilize the potential of all parameters in the pre-trained model. Therefore, it struggles to cover complex task spaces. Lack of expertise All inputs share the same adapter, making it difficult to model diversity for some complex or uncommon tasks; Unable to sparsify Each input token must traverse the same LoRA path, making the computation quite redundant. Inadequate generalization In some high-dimensional and complex tasks, low-rank constraints may limit the generalization ability of the model. In particular, the performance degradation is significant under multi-task and multi-modal conditions.
[0020] To address the four problems of LoRA mentioned above, this application proposes a model fine-tuning method. A hybrid expert architecture (MoE-LoRA) is added to the LoRA framework, introducing multiple LoRA modules into the target layer and dynamically selecting or combining appropriate experts based on the input through a gating network. This method not only overcomes the expressive limitations of a single LoRA module but also flexibly schedules different experts according to task characteristics, thus exhibiting stronger adaptability and generalization ability in complex task scenarios.
[0021] The following is in conjunction with the appendix Figures 2 to 8 As shown, the model fine-tuning method, apparatus and electronic device provided in this application are described in detail through specific embodiments and application scenarios.
[0022] like Figure 2 The diagram shown is a flowchart of a model fine-tuning method provided in an embodiment of this application. Figure 2 As shown, the model fine-tuning method may include the contents shown in steps 201 to 203.
[0023] In S201, a pre-trained model is obtained, and a parameter fine-tuning module is set in the target layer of the pre-trained model. The parameter fine-tuning module includes multiple LoRA modules and a gating module. The gating module is used to select a preset number of LoRA modules, with each LoRA module acting as an expert.
[0024] The pre-trained model can be a Transformer-based model. Transformer structures can be categorized into three types: Encoder-Decoder, Encoder-only, and Decoder-only. The pre-trained model can choose any of these structures; this embodiment does not impose any limitation. The target layer refers to a fine-tunable layer.
[0025] In one example, taking a large model of the decoder structure as an example, its structure is as follows: Figure 3 As shown.
[0026] The model includes an embedding layer, a positional encoding layer, a normalization layer, a self-attention layer, a feedforward / backward layer, and an output layer. The target layer can be either a self-attention layer or a feedforward / backward layer, depending on the specific application; this embodiment does not impose any limitations.
[0027] In one example, using the self-attention layer as the target layer, the structure of the self-attention layer can be as follows: Figure 4 As shown.
[0028] Where Q (Query), K (Key), and V (Value) are three matrices of uniform scale, for example, a 6x9 matrix: [[0.548, 0.912, 0.103, 0.765, 0.334, 0.881, 0.426, 0.677, 0.209], [0.991, 0.052, 0.637, 0.294, 0.803, 0.156, 0.742, 0.388, 0.511], [0.223, 0.776, 0.450, 0.128, 0.669, 0.999, 0.031, 0.584, 0.837], [0.604, 0.319, 0.872, 0.555, 0.241, 0.703, 0.492, 0.926, 0.188], [0.357, 0.680, 0.266, 0.943, 0.115, 0.579, 0.805, 0.442, 0.733], [0.162, 0.824, 0.539, 0.376, 0.690, 0.218, 0.951, 0.067, 0.614]] The inputs to these three matrices are linear variations of the output of the previous normalization layer. The formula is: Q=LayerNorm out *W Q K = LayerNorm out *W K V=LayerNorm out *W V The output is the self-attention function, and its formula is shown below:
[0029] Where, d kK represents the vector dimension of K. T This indicates that K is transposed, Softmax is the normalized exponential function, and Attention represents the self-attention output.
[0030] It is worth noting that if Q, K, and V are all 3*2 matrices, the size of the Attention function will also be 3*2 after calculation.
[0031] LoRA allows fine-tuning of the self-attention layer's parameters: Q, K, V, and O (output). These fine-tuned parameters are typically called projections, denoted by lowercase letters: q_proj (Query projection), k_proj (Key projection), v_proj (Value projection), and o_proj (output projection).
[0032] In this embodiment, multiple LoRA modules are introduced in the target layer (such as the self-attention layer). Each module acts as an expert, and a gate module is set up to obtain the gate value based on the input features. A portion of the experts are selected, for example, two experts (top 2) can be selected. Then, the expert outputs are weighted and used as the final ΔW.
[0033] In S202, the parameters of the target layer are fine-tuned based on the input of the target layer, the model preset configuration, and the parameters of the parameter fine-tuning module, to obtain the fine-tuned output of the target layer.
[0034] The input to the target layer can be a linear variation of the output of the previous normalization layer. The model's preset configuration can include at least the number of experts, the preset number (i.e., Top-K), the rank of LoRA, the scaling factor of LoRA, and the scale factor of LoRA. The parameters of the parameter fine-tuning module can include the parameters of the gating module and the parameters of the LoRA module (i.e., a pair of A and B matrices for each expert).
[0035] This embodiment achieves the same effect as adjusting the overall model parameters by fine-tuning the parameters of the target layer, thus reducing the overhead of training parameters.
[0036] In S203, the fine-tuned model is obtained based on the fine-tuned output and the base output of the pre-trained model. The base output is the output corresponding to the parameters frozen in the pre-trained model.
[0037] In this embodiment, a pre-trained model is first obtained, and a parameter fine-tuning module is set in the target layer of the pre-trained model. The parameter fine-tuning module includes multiple LoRA modules and a gating module. The gating module selects a preset number of LoRA modules, each acting as an expert. Then, based on the input of the target layer, the model's preset configuration, and the parameters of the parameter fine-tuning module, the parameters of the target layer are fine-tuned to obtain the fine-tuned output of the target layer. Finally, based on the fine-tuned output and the base output of the pre-trained model, a fine-tuned model is obtained. The base output corresponds to the output of the frozen parameters in the pre-trained model. This embodiment sets multiple LoRA modules and a gating module in the target layer of the pre-trained model, and uses the gating module to select a preset number of LoRA modules. This means the gating module can select a preset number of experts based on the input, enabling the model to overcome the limitations of a single LoRA module's expressive power. By flexibly scheduling different experts, the model can have stronger adaptability in complex task scenarios.
[0038] In one possible implementation of this application, the parameters of the target layer are fine-tuned based on the input of the target layer, the model preset configuration, and the parameters of the parameter fine-tuning module to obtain the fine-tuned output of the target layer. This may include: initializing the parameters of the parameter fine-tuning module and the parameters frozen in the pre-trained model; and, while keeping the parameters frozen in the pre-trained model unchanged, fine-tuning the parameters of the initialized parameter fine-tuning module using the input of the target layer and the model preset configuration to obtain the fine-tuned output of the target layer.
[0039] In this embodiment, the parameters are first initialized, including the frozen parameters and the parameters of the parameter fine-tuning module. Since most of the model's parameters are frozen, only a pair of A and B matrices for each expert needs to be trained, while other parts remain unchanged; that is, the frozen parameters remain unchanged. Using the input and the model's preset configuration, the A and B matrices for each expert are trained to obtain the final output. This embodiment utilizes the characteristics of LoRA, requiring only low-rank training, thus reducing the computational load.
[0040] In one possible implementation of this application, initializing the parameters of the parameter fine-tuning module and the frozen parameters in the pre-trained model may include: initializing the gate parameters in the parameter fine-tuning module, the gate parameters including gate weights and gate biases; initializing the LoRA parameters in the parameter fine-tuning module, the LoRA parameters including a first low-rank matrix and a second low-rank matrix, the rank of the product of the first low-rank matrix and the second low-rank matrix being the same as the rank of the weight matrix of the pre-trained model, and the LoRA module and LoRA parameters being in one-to-one correspondence; and initializing the basis weights in the frozen parameters in the pre-trained model.
[0041] Each expert has a pair of A and B matrices, namely a first low-rank matrix and a second low-rank matrix. The size of the first low-rank matrix is the product of the dimension of the input feature and the rank of LoRA, and the size of the second low-rank matrix is the product of the rank of LoRA and the dimension of the output feature.
[0042] This embodiment ensures the stability of the pre-trained weights in the early stages of fine-tuning by initializing each parameter, while gradually introducing low-rank adjustments to avoid training instability or performance degradation.
[0043] In one possible implementation of this application, with the parameters frozen in the pre-trained model remaining unchanged, the parameters of the initialized parameter fine-tuning module are fine-tuned using the input of the target layer and the model's preset configuration to obtain the fine-tuned output of the target layer. This can include: calculating the score of each expert using the input of the target layer, the gate weights, the gate bias, and the first and second low-rank matrices of each LoRA parameter; determining the sparse probability of the input of the target layer based on the score of each expert; calculating the incremental output of the input of the target layer after passing through each expert; and obtaining the fine-tuned output of the target layer based on the sparse probability and the incremental output.
[0044] In this embodiment, a preset number of experts are selected by calculating the affinity score (logits) of each expert, that is, the preset number of experts with the highest scores are selected. Then, the incremental output of the input after passing through each expert is calculated, and the fine-tuning output of the target layer is determined. The fine-tuning output refers to the output of only training the A and B parameter parts of the LoRA module.
[0045] In one possible implementation of this application, obtaining a fine-tuned model based on the fine-tuned output and the base output of the pre-trained model may include: determining the base output of the pre-trained model based on the input of the target layer and the frozen parameters in the initialized pre-trained model; adding the base output and the fine-tuned output to obtain the weight matrix of the pre-trained model; and determining the fine-tuned model based on the weight matrix.
[0046] In addition to the fine-tuned outputs mentioned above, the model outputs also include the outputs of the frozen portion, i.e., the outputs of the unadjusted portion. This output is the product of the input and the frozen weight parameters. The fine-tuned outputs and the frozen outputs are added together to obtain the weight matrix. Finally, the final fine-tuned model is obtained based on the weight matrix and the model input.
[0047] like Figure 5 The diagram shown is a schematic representation of the overall structure.
[0048] Among them, the gating module and multiple LoRA modules form the parameter fine-tuning module of the target layer. Through this module, the fine-tuning output ΔW can be obtained, the output of the frozen part is W0, and the weight matrix is W.
[0049] In one possible implementation of this application, the model fine-tuning method may further include: using the sum of the cross-entropy loss function and the expert equilibrium loss function as the loss function of the pre-trained model to fine-tune the parameters of the target layer, wherein the expert equilibrium loss function is used to balance the utilization rate of each expert.
[0050] The cross-entropy loss function is shown in the following equation:
[0051] Where y is the actual output. This is the predicted output.
[0052] To ensure uniformity in expert usage and avoid the continuous use of certain experts while others remain unused, this embodiment adds an auxiliary loss function, as shown below:
[0053] Where lb_coef is the load balancing coefficient, which is set to 0.01 here, m is the number of experts, and P is the average probability of the threshold value.
[0054] The loss function of the pre-trained model is shown below: loss=L(y, )+aux_loss In this embodiment of the application, by setting an expert equilibrium loss function, the situation where some experts are always used while others are never used can be avoided.
[0055] In one example, fine-tuning q_proj is used as an illustration (the process of fine-tuning k_proj, v_proj, and o_proj is similar to that of q_proj, and will not be repeated in the embodiments of this application).
[0056] Taking input x with a size of 2*3 as an example, x has 2 tokens, and each token has a size of 3 (in_features): x = [[1.0, 2.0, 3.0],# Token 1 [4.0, 5.0, 6.0]# Token 2 The model's preset configuration is set as follows: Number of experts: num_experts = 4; Top-K: top_k = 2; The rank of LoRA is r = 2. LoRA scaling factor: lora_alpha = 2; LoRA scaling factor: scale=1.0 (scale = lora_alpha / r = 1.0).
[0057] Initialization parameters, including frozen parameters, gating parameters, and LoRA parameters, are as follows: The q_proj layer (freezing weight parameters) includes base.weight (base weights) and base.bias (base bias). In this embodiment, base.bias is removed (and also removed for k_proj, v_proj, and o_proj) to speed up model convergence, retaining only the weight parameters of base.weight. The size of base.weight is 2*3, as shown below: base.weight = [[1, 1, 1], [2, 2, 2]].
[0058] The gate layer consists of `gate.weight` (gating weight) and `gate.bias` (gating bias). `gate.weight` has a size of 3x4, and `gate.bias` has a size of 1x4, as shown below: gate.weight = [[ 1, -1, 1, -1], # (in_features x num_experts represents the product of the input feature dimension and the number of experts, i.e., gate.weight) [ 1, -1,1, -1], [ 1, -1,1, -1]] gate.bias = [0,0,1,1].
[0059] There are 4 LoRA modules (experts), each expert has a pair of A and B matrices (ΔW=A). B).
[0060] Matrix A has a size of 3*2 (in_features x r), and matrix B has a size of 2*2 (r x out_features). Among them, Expert 1 is shown below: A1 = [[1,0],[0,1],[0,0]];B1 = [[1,1],[1,1]] Expert 2 is shown below: A2 = [[0,0],[1,0],[0,1]];B2 = [[2,2],[2,2]] Expert 3 is shown below: A3 = [[0,1],[0,0],[1,0]];B3 = [[3,3],[3,3]] Expert 4 is shown below: A4 = [[1,0],[1,0],[1,0]]; B4 = [[4,4],[4,4]].
[0061] The base_out of the pre-trained model is shown below: That is, the original, unmodified model calculation, i.e., W0: base_out = x @ base.weight.T (where T represents the transpose of weight) Token 1: [1, 2, 3] @ [[1, 2], [1, 2], [1, 2]]= [1*1+2*1+3*1, 1*2+2*2+3*2] = [6, 12] Token 2: [4, 5, 6] @ [[1, 2], [1, 2], [1, 2]]= [4*1+5*1+6*1, 4*2+5*2+6*2] = [15, 30] base_out = [[6, 12], [15, 30]].
[0062] The scores for each expert are calculated as follows: That is, calculate the affinity score (logits) of each token to the four experts, and then select the two with the highest affinity scores.
[0063] logits = x@gate.weight + gate.bias Token 1: [1,2,3] @ [[1,-1,1,-1],[1,-1,1,-1],[1,-1,1,-1]]+ [0,0,1,1] = [6, -6, 6, -6] + [0, 0, 1, 1] = [6, -6, 7, -5] Token 2: [4,5,6] @ [[1,-1,1,-1],[1,-1,1,-1],[1,-1,1,-1]]+ [0,0,1,1] = [15, -15, 15, -15] + [0, 0, 1, 1] = [15, -15, 16, -14] logits = [[6, -6, 7, -5], [15, -15, 16, -14]] Apply softmax to each row of logits to obtain the probability, and select the Top-2.
[0064] Token 1: logits = [6, -6, 7, -5]. The largest is the 3rd one (7), and the second largest is the 1st one (6).
[0065] Top-2 index: [2, 0] (Expert 3 and Expert 1) After softmax, the sparse probability (sparse_prob) results are: [0.26, 0.00, 0.72, 0.00] Token 2: logits = [15, -15, 16, -14]. The largest is the 3rd one (16), and the second largest is the 1st one (15).
[0066] Top-2 index: [2, 0] (Expert 3 and Expert 1) After softmax, the sparse probability (sparse_prob) results are: [0.26, 0.00, 0.72, 0.00] Finally, the sparse probabilities (sparse_probs) of Token 1 and Token 2 are as follows: [[0.26, 0.00, 0.72, 0.00], [0.26, 0.00, 0.72, 0.00]].
[0067] Calculate the increment ΔW*x for all experts.
[0068] That is, calculate the incremental output of each token after going through all four expert meetings, the formula is delta = (x @A) @ B.
[0069] For Token 1 = [1, 2, 3]: delta1: ([1,2,3] @ A1) @ B1 = ([1,2]) @ B1 = [3, 3] delta2: ([1,2,3] @ A2) @ B2 = ([2,3]) @ B2 = [10, 10] delta3: ([1,2,3] @ A3) @ B3 = ([3,1]) @ B3 = [12, 12] delta4: ([1,2,3] @ A4) @ B4 = ([6,0]) @ B4 = [24, 24] That is, the delta_all of Token 1: [[3,3], [10,10], [12,12], [24,24]] For Token 2 = [4, 5, 6]: delta1: ([4,5,6] @ A1) @ B1 = ([4,5]) @ B1 = [9, 9] delta2: ([4,5,6] @ A2) @ B2 = ([5,6]) @ B2 = [22, 22] delta3: ([4,5,6] @ A3) @ B3 = ([6,4]) @ B3 = [30, 30] delta4: ([4,5,6] @ A4) @ B4 = ([15,0]) @ B4 = [60, 60] That is, the delta_all of Token 2: [[9,9], [22,22], [30,30], [60,60]].
[0070] The final increment, weighted, is obtained by weighting the delta_all obtained in the above embodiment using sparse_probs as weights.
[0071] Among them, Token 1: weighted1 = 0.26 * [3,3] + 0.00 * [10,10] + 0.72 * [12,12] + 0.00 *[24,24] = [0.78, 0.78] + [8.64, 8.64] = [9.42, 9.42] Token 2: weighted2 = 0.26 * [9,9] + 0.00 * [22,22] + 0.72 * [30,30] + 0.00 *[60,60] = [2.34, 2.34] + [21.6, 21.6] = [23.94, 23.94] That is, weighted = [[9.42, 9.42], [23.94, 23.94]].
[0072] Finally, the output of the base path is added to the weighted increment, i.e., final_output = base_out + scale * weighted.
[0073] Wherein, Token 1: [6, 12] + [9.42, 9.42] = [15.42, 21.42] Token 2: [15, 30] + [23.94, 23.94]= [38.94, 53.94] Finally, the output of the q_proj layer is: final_output = [[15.42, 21.42], [38.94, 53.94]].
[0074] In one specific embodiment of this application, the training data includes audio recommendation data, totaling 46,972 entries, and the training data template can use alpaca_prompt as the template.
[0075] The training data example is shown below: "uid": "1687478xxx", "instruction": "You are an audio recommendation expert, skilled at recommending similar audio to users based on their historical audio listening tags." Input: "[User ID: 1687478xxx] is a user with [Gender: Female], [Age: 27], and [City Code: 100x]. Her top 3 most recently listened-to audio tags are: 1. Daily Log 2. Today's Outfit (ootd) 3. [Brand Name] Based on the above information, please provide 3 audio topics she is most likely to want to hear." Output: 1. Recording everyday moments\n2. Sharing fashion outfits\n3. xx brand and its founder", "history": [] The training parameters are as follows: r: int = 16, # Rank of LoRA lora_alpha: int = 16, # LoRA scaling factor alpha lora_dropout: float = 0.05, # LoRA's random dropout probability num_experts: int = 4, # Number of experts top_k: int = 2, # Number of experts selected remove_bias: bool = True, # Removes the bias term from the target layer lb_coef: float = 0.01, # Load balancing loss coefficient per_device_train_batch_size: =2 # Number of training batches gradient_accumulation_steps: =4 # Accumulated gradient values learning_rate: =1.0e-4 # Learning rate num_train_epochs:=3.0 # Number of training epochs Based on the above training examples and training parameters, the training results can be obtained as follows: Figure 6 As shown.
[0076] In one specific embodiment, an example of audio topic recommendation is shown below: Model input: <|im_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|> <|im_start|>user You are an audio recommendation expert, skilled at recommending similar audio to users based on their historical audio listening tags.
[0077] User ID: 1011861xxx is a user with the following characteristics: Female, Age: 26, City Code: 3.
[0078] The top 3 audio tags I've been listening to recently are: 1. Character 1 2. Character 2 3. Character 3 Based on the information above, please output three audio topics she is most likely to want to hear. Model output: <|im_start|>assistant 1. Comparison of Person 1's mindset with that of the xx group of people. 2. Character 2's Growth Process 3. Commemorating Person 3<|im_end|>.
[0079] In this embodiment, multiple LoRA modules are introduced into the target layer, and appropriate experts are dynamically selected or combined according to the input through a gating network. This enables the model to not only overcome the limitations of the expressive power of a single LoRA, but also to flexibly schedule different experts according to the task characteristics, thereby exhibiting stronger adaptability and generalization ability in complex task scenarios.
[0080] This application also provides a model fine-tuning device. For example... Figure 7 The diagram shown is a structural schematic of a model fine-tuning device provided in an embodiment of this application. Figure 7 As shown, the model fine-tuning device may include: an acquisition module 601, a fine-tuning module 602, and a determination module 603.
[0081] The acquisition module 201 is used to acquire a pre-trained model and set a parameter fine-tuning module in the target layer of the pre-trained model. The parameter fine-tuning module includes multiple LoRA modules and a gating module. The gating module is used to select a preset number of LoRA modules, with each LoRA module acting as an expert. The fine-tuning module 602 is used to fine-tune the parameters of the target layer based on the input of the target layer, the preset configuration of the model, and the parameters of the parameter fine-tuning module to obtain the fine-tuned output of the target layer. The determination module 603 is used to obtain the fine-tuned model based on the fine-tuned output and the basic output of the pre-trained model. The basic output is the output corresponding to the frozen parameters in the pre-trained model.
[0082] In this embodiment, module 201 first acquires a pre-trained model and sets a parameter fine-tuning module in the target layer of the pre-trained model. The parameter fine-tuning module includes multiple LoRA modules and a gating module. The gating module selects a preset number of LoRA modules, each acting as an expert. Then, fine-tuning module 602 fine-tunes the parameters of the target layer based on the input of the target layer, the model's preset configuration, and the parameters of the parameter fine-tuning module, obtaining the fine-tuned output of the target layer. Finally, determining module 603 determines the fine-tuned model based on the fine-tuned output and the base output of the pre-trained model. The base output corresponds to the output of the frozen parameters in the pre-trained model. This embodiment sets multiple LoRA modules and a gating module in the target layer of the pre-trained model, using the gating module to select a preset number of LoRA modules. This means the gating module can select a preset number of experts based on the input, enabling the model to overcome the limitations of a single LoRA module's expressive power. By flexibly scheduling different experts, the model can have stronger adaptability in complex task scenarios.
[0083] In one possible implementation of this application, the fine-tuning module 602 is used to: initialize the parameters of the parameter fine-tuning module and the parameters frozen in the pre-trained model; while keeping the parameters frozen in the pre-trained model unchanged, fine-tune the parameters of the initialized parameter fine-tuning module using the input of the target layer and the model preset configuration to obtain the fine-tuned output of the target layer.
[0084] In one possible implementation of this application, the fine-tuning module 602 is used to: initialize the gating parameters in the parameter fine-tuning module, the gating parameters including gating weights and gating biases; initialize the LoRA parameters in the parameter fine-tuning module, the LoRA parameters including a first low-rank matrix and a second low-rank matrix, the rank of the product of the first low-rank matrix and the second low-rank matrix is the same as the rank of the weight matrix of the pre-trained model, and the LoRA module corresponds one-to-one with the LoRA parameters; and initialize the basis weights in the frozen parameters in the pre-trained model.
[0085] In one possible implementation of this application, the fine-tuning module 602 is configured to: calculate the score of each expert using the input of the target layer, the gate weights, the gate bias, and the first low-rank matrix and the second low-rank matrix of each LoRA parameter; determine the sparse probability of the input of the target layer based on the score of each expert; calculate the incremental output of the input of the target layer after passing through each expert; and obtain the fine-tuned output of the target layer based on the sparse probability and the incremental output.
[0086] In one possible implementation of this application, the determining module 603 is used to: determine the basic output of the pre-trained model based on the input of the target layer and the parameters frozen in the initialized pre-trained model; add the basic output and the fine-tuning output to obtain the weight matrix of the pre-trained model; and determine the fine-tuning model based on the weight matrix.
[0087] In one possible implementation of this application, the model fine-tuning device may further include a second determining module.
[0088] The second determining module is used to use the sum of the cross-entropy loss function and the expert equilibrium loss function as the loss function of the pre-trained model to fine-tune the parameters of the target layer. The expert equilibrium loss function is used to balance the utilization rate of each expert.
[0089] The model fine-tuning device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.
[0090] The model fine-tuning device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.
[0091] The model fine-tuning device provided in this application embodiment can achieve... Figures 2 to 6 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.
[0092] Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to various embodiments of this application. The electronic device 700 includes, but is not limited to, components such as: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, a processor 710, and a power supply 711. Those skilled in the art will understand that... Figure 8 The electronic device structures shown are not intended to limit the electronic device. An electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. In the embodiments of this application, the electronic device includes, but is not limited to, mobile phones, tablets, laptops, PDAs, in-vehicle terminals, wearable devices, and pedometers.
[0093] The processor 710 can be used to: acquire a pre-trained model and set a parameter fine-tuning module in the target layer of the pre-trained model; wherein the parameter fine-tuning module includes multiple LoRA modules and a gating module, the gating module is used to select a preset number of LoRA modules, each LoRA module acting as an expert; fine-tune the parameters of the target layer based on the input of the target layer, the preset configuration of the model, and the parameters of the parameter fine-tuning module to obtain the fine-tuned output of the target layer; and obtain the fine-tuned model based on the fine-tuned output and the basic output of the pre-trained model, wherein the basic output is the output corresponding to the frozen parameters in the pre-trained model.
[0094] In this embodiment, a pre-trained model is first obtained, and a parameter fine-tuning module is set in the target layer of the pre-trained model. The parameter fine-tuning module includes multiple LoRA modules and a gating module. The gating module selects a preset number of LoRA modules, each acting as an expert. Then, based on the input of the target layer, the model's preset configuration, and the parameters of the parameter fine-tuning module, the parameters of the target layer are fine-tuned to obtain the fine-tuned output of the target layer. Finally, based on the fine-tuned output and the base output of the pre-trained model, a fine-tuned model is obtained. The base output corresponds to the output of the frozen parameters in the pre-trained model. This embodiment sets multiple LoRA modules and a gating module in the target layer of the pre-trained model, and uses the gating module to select a preset number of LoRA modules. This means the gating module can select a preset number of experts based on the input, enabling the model to overcome the limitations of a single LoRA module's expressive power. By flexibly scheduling different experts, the model can have stronger adaptability in complex task scenarios.
[0095] It should be understood that, in this embodiment, the radio frequency unit 701 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink data from the base station and processes it with the processor 710; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 701 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc. Furthermore, the radio frequency unit 701 can also communicate with networks and other devices through a wireless communication system.
[0096] Electronic devices provide users with wireless broadband internet access through network module 702, such as helping users send and receive emails, browse web pages, and access streaming media.
[0097] The audio output unit 703 can convert audio data received by the radio frequency unit 701 or the network module 702 or stored in the memory 709 into audio signals and output them as sound. Furthermore, the audio output unit 703 can also provide audio output related to specific functions performed by the electronic device 700 (e.g., call signal reception sound, message reception sound, etc.). The audio output unit 703 includes a speaker, a buzzer, and a receiver, etc.
[0098] Input unit 704 is used to receive audio or video signals. Input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on display unit 706. The image frames processed by GPU 7041 can be stored in memory 709 (or other storage media) or transmitted via radio frequency unit 701 or network module 702. Microphone 7042 can receive sound and process such sound into audio data. The processed audio data can be converted into a format that can be transmitted to a mobile communication base station via radio frequency unit 701 in telephone call mode.
[0099] The electronic device 700 also includes at least one sensor 705, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 7061 according to the ambient light level, and the proximity sensor can turn off the display panel 7061 and / or backlight when the electronic device 700 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used to identify the posture of the electronic device (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. The sensor 705 may also include a fingerprint sensor, pressure sensor, iris sensor, molecular sensor, gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc., which will not be described in detail here.
[0100] The display unit 706 is used to display information input by the user or information provided to the user. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.
[0101] User input unit 707 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of electronic devices. Specifically, user input unit 707 includes a touch panel 7071 and other input devices 7072. Touch panel 7071, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 7071). Touch panel 7071 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 710, which receives and executes commands from the processor 710. In addition, touch panel 7071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to touch panel 7071, user input unit 707 may also include other input devices 7072. Specifically, other input devices 7072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.
[0102] Furthermore, the touch panel 7071 can cover the display panel 7061. When the touch panel 7071 detects a touch operation on or near it, it transmits the information to the processor 710 to determine the type of touch event. Subsequently, the processor 710 provides corresponding visual output on the display panel 7061 based on the type of touch event. Although in Figure 8 In this embodiment, the touch panel 7071 and the display panel 7061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 7071 and the display panel 7061 can be integrated to realize the input and output functions of the electronic device. The specific implementation is not limited here.
[0103] Interface unit 708 serves as an interface for connecting external devices to electronic device 700. For example, external devices may include a wired or wireless headphone port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 708 can be used to receive input from external devices (e.g., data, power, etc.) and transmit the received input to one or more components within electronic device 700, or it can be used to transmit data between electronic device 700 and external devices.
[0104] The memory 709 can be used to store software programs and various data. The memory 709 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 709 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0105] The processor 710 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 709, and by calling data stored in the memory 709, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. The processor 710 may include one or more processing units; preferably, the processor 710 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 710.
[0106] The electronic device 700 may also include a power supply 711 (such as a battery) that supplies power to various components. Preferably, the power supply 711 is logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.
[0107] In addition, the electronic device 700 includes some functional modules not shown, which will not be described in detail here.
[0108] This application also provides an electronic device, including a processor 710, a memory 709, and a computer program stored in the memory 709 and executable on the processor 710. When the computer program is executed by the processor 710, it implements the various processes of the above-described model fine-tuning method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.
[0109] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described model fine-tuning method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0110] This application provides a computer program product stored in a computer-readable storage medium. The computer program product is executed by at least one processor to implement the various processes of the above-described model fine-tuning method embodiments and achieve the same technical effects. To avoid repetition, it will not be described again here.
[0111] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described model fine-tuning method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.
[0112] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0113] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0115] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A model fine-tuning method, characterized in that, include: A pre-trained model is obtained, and a parameter fine-tuning module is set in the target layer of the pre-trained model; wherein, the parameter fine-tuning module includes multiple LoRA modules and a gating module, and the gating module is used to select a preset number of LoRA modules, each of the LoRA modules serving as an expert; Based on the input of the target layer, the model preset configuration, and the parameters of the parameter fine-tuning module, the parameters of the target layer are fine-tuned to obtain the fine-tuned output of the target layer; Based on the fine-tuned output and the base output of the pre-trained model, a fine-tuned model is obtained, wherein the base output is the output corresponding to the frozen parameters in the pre-trained model.
2. The method according to claim 1, characterized in that, The step of fine-tuning the parameters of the target layer based on the input of the target layer, the model preset configuration, and the parameters of the parameter fine-tuning module, to obtain the fine-tuned output of the target layer, includes: Initialize the parameters of the parameter fine-tuning module and the parameters frozen in the pre-trained model; With the parameters frozen in the pre-trained model remaining unchanged, the parameters of the parameter fine-tuning module are fine-tuned using the input of the target layer and the model preset configuration to obtain the fine-tuned output of the target layer.
3. The method according to claim 2, characterized in that, The initialization of the parameters of the parameter fine-tuning module and the parameters frozen in the pre-trained model includes: Initialize the gate parameters in the parameter fine-tuning module, the gate parameters including gate weights and gate biases; Initialize the LoRA parameters in the parameter fine-tuning module. The LoRA parameters include a first low-rank matrix and a second low-rank matrix. The rank of the product of the first low-rank matrix and the second low-rank matrix is the same as the rank of the weight matrix of the pre-trained model. The LoRA module corresponds one-to-one with the LoRA parameters. Initialize the basis weights in the frozen parameters of the pre-trained model.
4. The method according to claim 3, characterized in that, With the parameters frozen in the pre-trained model remaining unchanged, the parameters of the initialized parameter fine-tuning module are fine-tuned using the input of the target layer and the model's preset configuration to obtain the fine-tuned output of the target layer, including: The score for each expert is calculated using the input of the target layer, the gate weights, the gate bias, and the first low-rank matrix and the second low-rank matrix of each LoRA parameter. Based on the score of each expert, the sparsity probability of the input to the target layer is determined; The input to the target layer is calculated, followed by the incremental output of each expert. Based on the sparse probability and the incremental output, the fine-tuning output of the target layer is obtained.
5. The method according to claim 2, characterized in that, The process of obtaining the fine-tuned model based on the fine-tuned output and the base output of the pre-trained model includes: Based on the input of the target layer and the parameters frozen in the initialized pre-trained model, determine the basic output of the pre-trained model; The weight matrix of the pre-trained model is obtained by adding the base output and the fine-tuning output. Based on the weight matrix, the fine-tuning model is determined.
6. The method according to claim 1, characterized in that, The method further includes: The sum of the cross-entropy loss function and the expert equilibrium loss function is used as the loss function of the pre-trained model to fine-tune the parameters of the target layer. The expert equilibrium loss function is used to balance the utilization rate of each expert.
7. A model fine-tuning device, characterized in that, include: An acquisition module is used to acquire a pre-trained model and set a parameter fine-tuning module in the target layer of the pre-trained model; wherein, the parameter fine-tuning module includes multiple LoRA modules and a gating module, and the gating module is used to select a preset number of LoRA modules, each of the LoRA modules serving as an expert; The fine-tuning module is used to fine-tune the parameters of the target layer based on the input of the target layer, the model preset configuration, and the parameters of the parameter fine-tuning module, so as to obtain the fine-tuning output of the target layer; The determination module is used to obtain a fine-tuned model based on the fine-tuned output and the base output of the pre-trained model, wherein the base output is the output corresponding to the frozen parameters in the pre-trained model.
8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, include: A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, The computer program product is stored in a computer-readable storage medium, and the computer program product is executed by at least one processor to implement the steps of the method as described in any one of claims 1 to 6.