Hybrid expert model training and merging method based on efficient fine tuning of parameters

Abstract

The invention discloses a hybrid expert model training and merging method based on efficient fine tuning of parameters. The method comprises the following specific steps: S1, firstly, training a base model; s2, performing extraction and initialization LoRA on a matrix principal component of a main network layer by using a singular value decomposition method, and leaving a residual error in an original matrix; s3, for a plurality of fields, training a plurality of LoRA branches; and S4, after training is completed, adding a gate control layer, taking a plurality of LoRAs as activation experts of the MoE, taking the residual matrix as a side branch, and training the gate control layer by using a small part of merged data. According to the method, the computing power cluster scale is remarkably reduced, the convergence efficiency can be remarkably improved through the singular value decomposition method, the final parameter scale is remarkably reduced due to the fact that the final MoE uses the LoRA structure, and the model effect is better.

Description

Technical Field

[0001] The present invention relates to the technical field of model training, and in particular to a hybrid expert model training merging method based on efficient parameter fine-tuning. Background Art

[0002] Mixed Experts Model (MoE): This model combines multiple experts (sub-models) to handle complex tasks. The model introduces a gating mechanism (gate) to dynamically select experts to activate after a given input, thereby improving the efficiency and performance of the model. Compared with dense networks, it has better results and higher reasoning efficiency at the same parameter scale.

[0003] Parameter-Efficient Fine-Tuning (PEFT): It is used to fine-tune pre-trained deep learning models to adapt to specific tasks or fields with limited computing resources and parameter updates. PEFT optimizes and adapts the model by updating a small number of parameters or introducing auxiliary modules without changing most of the model parameters. Among them, the most widely used method in the field of deep learning is the LoRA method, which mainly bypasses the network with a low-rank matrix of A*B, and can be fine-tuned with a small amount of training.

[0004] Model synthesis: mainly combines multiple models into one model through model integration, knowledge distillation, model fusion, etc.

[0005] However, in the prior art, the hybrid expert model training merging method based on efficient parameter fine-tuning has the following disadvantages:

[0006] 1. Existing MoE models usually need to be trained from scratch with the MoE structure, which has high requirements for computing power clusters and cannot be adapted to smaller computing clusters;

[0007] 2. LoRA converges slowly and is prone to overfitting;

[0008] 3. Existing model synthesis methods usually increase the number of model parameters and introduce additional training, but the overall effect on improving the model is limited. Summary of the invention

[0009] The present invention provides a hybrid expert model training merging method based on efficient parameter fine-tuning to solve the problems raised in the above background technology.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] The hybrid expert model training merging method based on efficient parameter fine-tuning includes the following specific steps:

[0012] Step S1, first train a base model;

[0013] Step S2, using the singular value decomposition method to extract the main components of the matrix of the main network layer to initialize LoRA, and leaving the residual in the original matrix;

[0014] Step S3, for multiple fields, training multiple LoRA branches;

[0015] Step S4, after the training is completed, add a gating layer to use multiple LoRA as activation experts of MoE, the residual matrix as a side branch, and use a small part of the merged data to train the gating layer.

[0016] As a further improvement of the present technical solution: in step S1, the network layer includes but is not limited to FFN and Attention.

[0017] As a further improvement of the present technical solution: in step S2, the multiple fields include but are not limited to code, mathematics, and industry knowledge.

[0018] As a further improvement of the present technical solution: Step S1, first train a base model, specifically: use a general data set D to train a basic large model M0.

[0019] As a further improvement of this technical solution: Step S2, the main network layer uses the singular value decomposition method to extract the main components of the matrix to initialize LoRA, and leave the residual in the original matrix, specifically:

[0020] For the specific layer Li in the basic large model M0, the singular value decomposition method is performed, and each layer is decomposed into the residual matrix Ri and the LoRA initialization matrix Ai, Bi.

[0021] As a further improvement of this technical solution: Step S3, for multiple fields, multiple LoRA branches are trained, specifically:

[0022] For different downstream tasks, different LoRA matrices Aij, Bij are trained to obtain task models M1 to MN.

[0023] As a further improvement of the technical solution: Step S4, after the training is completed, a gating layer is added to use multiple LoRA as activation experts of MoE, the residual matrix as a side branch, and a small part of the merged data is used to train the gating layer, specifically:

[0024] All downstream task fine-tuning models Mi are merged into MoE to perform gating layer fine-tuning. The output of each layer is:

[0025] Among them, g(ij) is the gate activation function, which is 1 for the activated expert model and 0 for the unactivated one.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention not only significantly reduces the scale of the computing power cluster, but also the singular value decomposition method can significantly improve the convergence efficiency. The final MoE uses the LoRA structure, and the final parameter scale is also significantly reduced, and the model effect is better.

[0028] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0030] Figure 1 A schematic diagram of the structure of the hybrid expert model training merging method based on efficient parameter fine-tuning proposed by the present invention;

[0031] Figure 2 This is a schematic diagram of the steps of the hybrid expert model training merging method based on efficient parameter fine-tuning proposed by the present invention. DETAILED DESCRIPTION

[0032] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention. The present invention is described in more detail by way of example with reference to the accompanying drawings in the following paragraphs. It should be noted that the accompanying drawings are all in a very simplified form and are not in precise proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.

[0033] See also Figure 1 In an embodiment of the present invention, a hybrid expert model training merging method based on efficient parameter fine-tuning includes the following specific steps:

[0034] Step S1, first training a base model, specifically: using a general data set D to train a basic large model M;

[0035] Step S2, use the singular value decomposition method to extract the main components of the matrix for the main network layers (FFN, Attention, etc.) to initialize LoRA, and leave the residual in the original matrix, specifically:

[0036] For a specific layer L in the base large model M0 i , perform singular value decomposition, and decompose each layer into a residual matrix R i And the LoRA initialization matrix A i , B i ;

[0037] Step S3, for multiple fields (code, mathematics, industry knowledge, etc.), train multiple LoRA branches, specifically:

[0038] For different downstream tasks, train different LoRA matrices A ij , B ij , get the task model M1 to M N ;

[0039] Step S4, after the training is completed, add a gating layer to use multiple LoRA as the activation experts of MoE, the residual matrix as the side branch, and use a small part of the merged data to train the gating layer, specifically:

[0040] Fine-tune the model M for all downstream tasks i MoE merging is performed to fine-tune the gated layer, and the output of each layer is:

[0041] Among them, g(ij) is the gate activation function, which is 1 for the activated expert model and 0 for the unactivated one.

[0042] The training process of the present invention is much smaller than the computing cluster and computing amount required for overall training of a MoE model of the same size; and the singular value decomposition method is used to initialize LORA to prevent overfitting and improve the efficiency of training. The MoE model synthesis method retains the effects of each model better than other methods, and the effect will be further improved after activating experts through MoE, which is better than the effect of a single task model.

[0043] In the present invention, LORA+MoE can significantly reduce the overall model scale and improve model efficiency and output speed.

[0044] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any ordinary technician in the industry can smoothly implement the present invention as shown in the drawings and described above. However, any equivalent changes, modifications and evolutions made by technicians familiar with the profession without departing from the scope of the technical solution of the present invention using the technical content disclosed above are all equivalent embodiments of the present invention. At the same time, any equivalent changes, modifications and evolutions made to the above embodiments based on the essential technology of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A hybrid expert model training merging method based on efficient parameter fine-tuning, characterized in that: The specific steps include: Step S1, first train a base model; Step S2, use the singular value decomposition method to extract the main components of the matrix of the main network layer to initialize LoRA, and leave the residual in the original matrix; Step S3, for multiple fields, training multiple LoRA branches; Step S4, after the training is completed, add a gating layer to use multiple LoRA as activation experts of MoE, the residual matrix as a side branch, and use a small part of the merged data to train the gating layer.

2. The hybrid expert model training merging method based on efficient parameter fine-tuning according to claim 1 is characterized in that: In step S1, the network layer includes but is not limited to FFN and Attention.

3. The hybrid expert model training merging method based on efficient parameter fine-tuning according to claim 2 is characterized in that: In step S2, the multiple fields include but are not limited to code, mathematics, and industry knowledge.

4. The hybrid expert model training merging method based on efficient parameter fine-tuning according to claim 3 is characterized in that: Step S1, first train a base model, specifically: use the general data set D to train a basic large model M0.

5. The hybrid expert model training merging method based on efficient parameter fine-tuning according to claim 4 is characterized in that: Step S2, use the singular value decomposition method to extract the main components of the matrix for the main network layer to initialize LoRA, and leave the residual in the original matrix, specifically: For a specific layer L in the base large model M0 i , perform singular value decomposition, and decompose each layer into a residual matrix R i And the LoRA initialization matrix A i , B i .

6. The hybrid expert model training merging method based on efficient parameter fine-tuning according to claim 5 is characterized in that: Step S3, for multiple fields, train multiple LoRA branches, specifically: For different downstream tasks, train different LoRA matrices A ij , B ij , get the task models M1 to M N .

7. The hybrid expert model training merging method based on efficient parameter fine-tuning according to claim 6 is characterized in that: Step S4, after the training is completed, add a gating layer to use multiple LoRA as the activation experts of MoE, the residual matrix as the side branch, and use a small part of the merged data to train the gating layer, specifically: Fine-tune the model M for all downstream tasks i MoE merging is performed to fine-tune the gated layer, and the output of each layer is: Among them, g(ij) is the gate activation function, which is 1 for the activated expert model and 0 for the unactivated one.

Images