Large language model fine tuning method based on LoRA improvement

Abstract

The invention discloses a large language model fine tuning method based on LoRA improvement, and the method comprises the steps: obtaining a weight file of a large model, and obtaining a training data set according to a needed vertical field and a downstream task; building a fine adjustment platform; the method comprises the following steps: acquiring an open source LoRA code, and modifying the open source LoRA code to obtain a modified code; and performing fine adjustment on the selected large model by utilizing the training data set and the modified code, and loading the output LoRA module onto a weight matrix of the original model after fine adjustment is completed, so that the large model has the capability of better completing specific downstream tasks. According to the method, the training parameter quantity of the LoRA in the fine tuning process is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of large language models, and in particular to a large language model fine-tuning method based on LoRA improvement. Background Art

[0002] In order to make the pre-trained language model have better performance in the vertical field, people need to update its trainable parameters through fine-tuning. However, if the above operations are performed on all parameters, it will place huge demands on the hardware equipment, and the training time will often be very long. For example, even a smaller large language model (such as LLAMa-7B, Yi-6B, etc.) has billions of trainable parameters, and fine-tuning them in full also requires hundreds of GB of video memory, which is unacceptable to most people. In order to reduce the fine-tuning overhead of large language models and reduce the number of parameters adjusted during training, people have proposed a variety of efficient parameter fine-tuning methods. LoRA is one of the most well-known efficient parameter fine-tuning methods. Its author assumes that the model weight matrix has a lower intrinsic rank during the adjustment process, and no longer directly fine-tunes the weight matrix, but freezes the parameters of the weight matrix so that its gradient is no longer updated, and instead trains the A and B matrices after low-rank decomposition, and the updated parameters are replaced by the product of B and A. Because the number of parameters contained in the A and B matrices is much smaller than the number of parameters in the weight matrix, it can speed up the training and greatly reduce the demand for video memory during fine-tuning. When the model is quantized, the video memory required for fine-tuning a model with billions of parameters can even be reduced to less than 10GB. And after greatly reducing the amount of training parameters, LoRA can still achieve results similar to full fine-tuning. After LoRA was proposed, various variant fine-tuning methods based on LoRA were invented, and KronA is one of them. The main difference between this method and LoRA is that it replaces the matrices A and B in LoRA with the Kronecker product of the two matrices; because the amount of parameters trained by KronA will also be much smaller than full fine-tuning, this method can also speed up training. And compared with LoRA, the product of matrix A and matrix B in KronA can maintain the same rank as the original matrix, achieving a better balance between rank and parameter quantity. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a large language model fine-tuning method based on LoRA improvement, which reduces the number of parameters trained by LoRA during the fine-tuning process.

[0004] To achieve the above object, the present invention provides a large language model fine-tuning method based on LoRA improvement, comprising:

[0005] Obtain the weight file of the large model and obtain the training data set based on the required vertical field and downstream tasks;

[0006] Build a fine-tuned platform;

[0007] Obtain an open source LoRA code, modify the open source LoRA code, and obtain a modified code;

[0008] The selected large model is fine-tuned using the training data set and the modified code. After the fine-tuning is completed, the output LoRA module is loaded onto the weight matrix of the original model, so that the large model has a better ability to complete specific downstream tasks.

[0009] Optionally, the open source LoRA code is modified to obtain the modified code including:

[0010] Decompose the weight matrix into a single matrix A and train only the parameters contained in matrix A;

[0011] The weight matrix is ​​updated using the product of matrix A and the transpose of matrix A itself or the Kronecker product and forward propagation is performed to obtain the modified code.

[0012] Optionally, the method of updating the weight matrix and forward propagating using the product of matrix A and the transpose of matrix A itself is:

[0013] W 0 +ΔW=W 0 +A T A

[0014] Y=W 0 x+ΔWx=W 0 x+A T Ax

[0015] Among them, W 0 represents the weight matrix; ΔW represents the updated parameter; A represents the training matrix; T represents the transposed symbol; Y represents the output vector; x represents the input vector.

[0016] Optionally, the method of updating the weight matrix and forward propagating using the Kronecker product of the matrix A and the transposed matrix A itself is:

[0017]

[0018] Among them, W 0 represents the weight matrix; ΔW represents the updated parameter; A represents the training matrix; T represents the transposed symbol; Y represents the output vector; x represents the input vector; represents the Kronecker product.

[0019] Technical effect of the present invention: The present invention discloses a large language model fine-tuning method based on the improvement of LoRA, which mainly includes two methods: LoRAS and LoRASK. In LoRAS, the matrix B in the original LoRA is discarded, and only the parameters of the matrix A are trained, and then re-parameterized by the product of the matrix A and its own transpose. LoRASK also discards the matrix B and only retains the matrix A, but it is re-parameterized by the Kronecker product of the matrix A and its own transpose. Compared with the original LoRA method and the KronA method, the technology provided by the present invention further reduces the amount of parameters in the training process. For example, when r=8, the amount of training parameters of LoRAS is reduced to half of that of LoRA, while the amount of training parameters of LoRASK is only one sixteenth of that of LoRA. Because this improvement further reduces the amount of parameters, it can better reduce the demand for device storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 This is a structural diagram of a large language model fine-tuning method based on LoRA improvement according to an embodiment of the present invention;

[0022] Figure 2 Schematic diagram of the score ratio (LoRAS / LoRA) of LoRAS and LoRA on a selected evaluation benchmark according to an embodiment of the present invention;

[0023] Figure 3 Schematic diagram of the score ratio (LoRASK / LoRA) of LoRASK and LoRA on a selected evaluation benchmark according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] like Figure 1 As shown, this embodiment provides a large language model fine-tuning method based on LoRA improvement, including: obtaining a weight file of the large model, and obtaining a training data set according to the required vertical field and downstream tasks;

[0027] Build a fine-tuned platform;

[0028] Obtain an open source LoRA code, modify the open source LoRA code, and obtain a modified code;

[0029] The selected large model is fine-tuned using the training data set and the modified code. After the fine-tuning is completed, the output LoRA module is loaded onto the weight matrix of the original model, so that the large model has a better ability to complete specific downstream tasks.

[0030] Furthermore, the open source LoRA code is modified to obtain the modified code including:

[0031] Decompose the weight matrix into a single matrix A and train only the parameters contained in matrix A;

[0032] The weight matrix is ​​updated using the product of matrix A and the transpose of matrix A itself or the Kronecker product and forward propagation is performed to obtain the modified code.

[0033] Furthermore, the method of updating the weight matrix and forward propagating using the product of matrix A and the transpose of matrix A itself is:

[0034] W 0 +ΔW=W 0 +A T A

[0035] Y=W 0 x+ΔWx=W 0 x+A T Ax

[0036] Among them, W 0 represents the weight matrix; ΔW represents the updated parameter; A represents the training matrix; T represents the transposed symbol; Y represents the output vector; x represents the input vector.

[0037] Furthermore, the method of updating the weight matrix and forward propagating using the Kronecker product of the matrix A and the transposed matrix A itself is:

[0038]

[0039] Among them, W 0 represents the weight matrix; ΔW represents the updated parameter; A represents the training matrix; T represents the transposed symbol; Y represents the output vector; x represents the input vector; represents the Kronecker product.

[0040] During the inference process of the model, by calculating and storing W = W 0 +A TA can be used for inference as usual, so no matter which method is used, there will be no additional inference delay. But it is worth noting that the product of a matrix and its transpose results in a matrix with the same number of rows and columns. However, for some models, or some self-attention modules of the same model, the number of rows and columns of their weight matrices are not equal. Therefore, they cannot be transformed by W=W 0 +A T A is used to update the weights. The method of the present invention cannot be applied to this type of model for the time being.

[0041] like Figure 1 The reparameterization process of our method is shown. Although the decomposition methods of LoRAS and LoRASK are different, they only train one matrix and reparameterize the weights by multiplying it with its own transpose or Kronecker product.

[0042] In order to verify the feasibility of LoRAS, three popular fine-tuning datasets were selected: Alpaca, Adgen, and WebQA. Then, the BLEU-4, ROUGE-1, ROUGE-2, and ROUGE-L scores of LoRA, LoRAS, and LoRASK on the above datasets were evaluated on the large language model Yi.

[0043] The experiment of this invention relies on a GitHub project called Llama Factory.

[0044] Before comparing LoRAS with LoRA, we first tested the effect of r value on the fine-tuning effect of LoRAS on the Alpaca dataset. Table 1 shows that as r increases, the fine-tuning effect is further enhanced. This phenomenon is not only in line with expectations, but also consistent with the law of LoRA.

[0045] Table 1

[0046] r BLEU-4 ROUGE-1 ROUGE-2 ROUGE-L 2 40.33 45.16 23.25 37.51 4 40.56 45.55 23.60 37.91 8 41.03 45.89 23.92 38.30 16 41.47 46.17 24.20 38.63

[0047] like Figure 2 The score ratio of LoRAS and LoRA is shown. It can be seen that although the training time of LoRAS is slightly longer when the number of training parameters is half of LoRA, its score is very close to LoRA. Therefore, it is feasible to discard the B matrix in LoRA and only train the A matrix.

[0048] In KronA, the ratio of the number of rows to the number of columns of A and B will have a certain impact on the fine-tuning effect. Therefore, before conducting a comparative experiment between LoRASK and LoRA, we first explored the relationship between the number of rows and the number of columns of A and B. 1 / d 2 The effect of the value of (the ratio of the number of rows to the number of columns of matrix A) on the fine-tuning results of LoRASK is shown in Table 2 for different d1 / d 2 The difference in scores of LoRASK on the Alpaca dataset is shown in the figure below. It can be seen that d 1 / d 2 The larger the value of , the better the fine-tuning effect. Therefore, in the comparison experiment between LoRASK and LoRA, we set d 1 / d 2 The value is set to a relatively large 64.

[0049] Table 2

[0050]

[0051]

[0052] like Figure 3 The score ratio of LoRASK and LoRA is shown. Compared with LoRAS, LoRASK's score has a larger gap with LoRA (but the gap in this score is mostly within 10%), and the training time is further extended. Since only a single matrix is ​​used for fine-tuning, and the Kronecker product is used instead of a simple low-rank decomposition, when r is set to 8, the number of training parameters of LoRASK is only one sixteenth of that of LoRA.

[0053] In comprehensive comparison, LoRAS performs better than LoRASK. This difference may come from the decomposition method of the weight matrix and the number of training parameters.

[0054] Based on the well-known parameter-efficient fine-tuning method LoRA, an improved fine-tuning method is proposed: LoRAS and LoRASK. Like LoRA, LoRAS and LoRASK freeze the parameters of the weight matrix and only train the decomposed matrix. But the improvement is that we discard the B matrix and only train a single matrix A, so that the number of fine-tuning parameters is reduced to half or even less than the original method. This experiment proves the feasibility and applicability of this improved method, and shows that in the LoRA method, fine-tuning of a large language model can be achieved by training only a single matrix.

[0055] The present invention discloses a large language model fine-tuning method based on LoRA improvement, which mainly includes two methods: LoRAS and LoRASK. In LoRAS, the matrix B in the original LoRA is discarded, and only the parameters of the matrix A are trained, and then re-parameterized by the product of the matrix A and its own transpose. LoRASK also discards the matrix B and only retains the matrix A, but it is re-parameterized by the Kronecker product of the matrix A and its own transpose. Compared with the original LoRA method and the KronA method, the technology provided by the present invention further reduces the amount of parameters in the training process. For example, when r=8, the amount of training parameters of LoRAS is reduced to half of that of LoRA, while the amount of training parameters of LoRASK is only one sixteenth of that of LoRA. Because this improvement further reduces the amount of parameters, it can better reduce the demand for device storage.

[0056] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A large language model fine-tuning method based on LoRA improvement, characterized in that: include: Obtain the weight file of the large model and obtain the training data set based on the required vertical field and downstream tasks; Build a fine-tuned platform; Obtain an open source LoRA code, modify the open source LoRA code, and obtain a modified code; The selected large model is fine-tuned using the training data set and the modified code. After the fine-tuning is completed, the output LoRA module is loaded onto the weight matrix of the original model, so that the large model has a better ability to complete specific downstream tasks.

2. The method for fine-tuning a large language model based on LoRA improvement as claimed in claim 1, characterized in that: The open source LoRA code is modified to obtain the modified code including: Decompose the weight matrix into a single matrix A and train only the parameters contained in matrix A; The weight matrix is ​​updated using the product of matrix A and the transpose of matrix A itself or the Kronecker product and forward propagation is performed to obtain the modified code.

3. The large language model fine-tuning method based on LoRA improvement as claimed in claim 2, characterized in that: The method of updating the weight matrix and forward propagating using the product of matrix A and the transpose of matrix A itself is: W0+ΔW=W0+A T THE Y=W0x+ΔWx=W0x+A T Ax Among them, W0 represents the weight matrix; ΔW represents the updated parameter; A represents the training matrix; T represents the transposed symbol; Y represents the output vector; x represents the input vector.

4. The method for fine-tuning a large language model based on LoRA improvement as claimed in claim 2, characterized in that: The method of updating the weight matrix and forward propagating using the Kronecker product of matrix A and the transpose of matrix A itself is: Where W0 represents the weight matrix; ΔW represents the updated parameter; A represents the training matrix; T represents the transposed symbol; Y represents the output vector; x represents the input vector; represents the Kronecker product.

Images