Large language model training method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a big language model training method and device, electronic equipment and a readable storage medium, and particularly relates to the technical field of artificial intelligence, and the method comprises the steps: determining task loss according to a prediction result corresponding to the current batch of training data of a big language model and a real label; determining regularization loss according to a singular value of a first low-rank matrix or a second low-rank matrix corresponding to a weight matrix of a target layer in the large language model; determining the total loss of the training data in the current batch according to the task loss and the regularization loss; updating parameters of the first low-rank matrix and the second low-rank matrix according to the total loss; and updating the target layer according to the first low-rank matrix and the second low-rank matrix under the condition that the large language model meets a preset training ending condition. The problem that in the prior art, the model complexity cannot be reduced, so that the model cannot pay attention to the main features of the specific field of the sample is solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and readable storage medium for training large language models. Background Technology

[0002] Large language models (such as the GPT series and BERT) perform exceptionally well in natural language processing tasks, but fine-tuning all parameters requires significant computational resources. When the model is fully trained and applied to samples from a specific domain—for example, a text processing model working with agricultural texts—its excessive complexity prevents it from focusing on the key features of the agricultural text.

[0003] It is evident that existing technologies cannot reduce model complexity, which prevents the model from focusing on the key features of the sample's specific domain rather than noise.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to solve the problem in the prior art that the model cannot reduce the model complexity, which leads to the model being unable to focus on the main features of the sample in a specific domain, and to provide a large language model training method, device, electronic device and readable storage medium.

[0006] The first aspect of this invention provides a method for training a large language model, comprising: determining a task loss based on the prediction results and ground truth labels corresponding to the training data of the current batch of the large language model; determining a regularization loss based on the singular values ​​of a first low-rank matrix or a second low-rank matrix corresponding to the weight matrix of the target layer in the large language model; determining the total loss of the training data in the current batch based on the task loss and the regularization loss; updating the parameters of the first low-rank matrix and the second low-rank matrix based on the total loss; and updating the target layer based on the first low-rank matrix and the second low-rank matrix when the large language model meets a preset training termination condition.

[0007] In one embodiment of the present invention, determining the regularization loss based on the singular values ​​of the low-rank matrix of the target layer in the large language model includes: performing singular value decomposition on the first low-rank matrix or the second low-rank matrix being trained in the current batch to obtain the singular values; and performing L2 regularization on the singular values ​​to obtain the regularization loss.

[0008] In one embodiment of the present invention, determining the total loss of the training data in the current batch based on the task loss and the regularization loss includes: performing a weighted summation of the task loss and the regularization loss to obtain the total loss of the training data in the current batch.

[0009] In one embodiment of the present invention, the gradients of the parameters of the first low-rank matrix and the second low-rank matrix are determined based on the total loss; the parameters of the first low-rank matrix and the second low-rank matrix are updated using a gradient descent algorithm; wherein the learning rate of the first low-rank matrix and the learning rate of the second low-rank matrix are set to different values.

[0010] In one embodiment of the present invention, the learning rate of the first low-rank matrix and the learning rate of the second low-rank matrix are set to different values.

[0011] In one embodiment of the present invention, updating the target layer based on the first low-rank matrix and the second low-rank matrix includes: determining an increment matrix based on the first low-rank matrix and the second low-rank matrix; and determining the final weights of the target layer based on the increment matrix and the original weights of the target layer.

[0012] In one embodiment of the present invention, determining the increment matrix based on the first low-rank matrix and the second low-rank matrix includes:

[0013] in, It is an incremental matrix. Let be the first low-rank matrix. This is the second low-rank matrix; The step of determining the final weights of the target layer based on the incremental matrix and the original weights of the target layer includes:

[0014] in, For the final weight, The original weights are given.

[0015] A second aspect of the present invention provides a large language model training apparatus, comprising: a first determining module, configured to determine a task loss based on the prediction results and ground truth labels corresponding to the training data of the current batch of the large language model; a second determining module, configured to determine a regularization loss based on the singular values ​​of a first low-rank matrix or a second low-rank matrix corresponding to the weight matrix of the target layer in the large language model; a third determining module, configured to determine the total loss of the training data in the current batch based on the task loss and the regularization loss; a first updating module, configured to update the parameters of the first low-rank matrix and the second low-rank matrix based on the total loss; and a second updating module, configured to update the target layer based on the first low-rank matrix and the second low-rank matrix when the large language model meets a preset training termination condition.

[0016] A third aspect of the present invention provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the large language model training method as described in the first aspect.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the large language model training method as described in the first aspect.

[0018] This invention determines the task loss based on the prediction results and true labels corresponding to the current batch of training data in a large language model; it determines the regularization loss based on the singular values ​​of the first or second low-rank matrix corresponding to the weight matrix of the target layer in the large language model; it determines the total loss of the training data in the current batch based on the task loss and the regularization loss; it updates the parameters of the first and second low-rank matrices based on the total loss; and it updates the target layer based on the first and second low-rank matrices when the large language model meets the preset training termination conditions. By calculating the regularization loss simultaneously with the task loss, and then determining the total loss based on the task loss and the regularization loss, the optimization process is guided to automatically and smoothly reduce the redundant complexity in the low-rank matrix while learning new tasks. This solves the problem in existing technologies where the inability to reduce model complexity leads to the model's inability to focus on the main features of a specific domain of the samples. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a large language model training method according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the framework of a large language model training device according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the frame of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0021] The technical solution of the present invention is illustrated below through specific embodiments. It should be understood that the one or more steps mentioned in the present invention do not preclude the existence of other methods and steps before or after the combined steps, or that other methods and steps may be inserted between these explicitly mentioned steps. It should also be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the present invention. Unless otherwise stated, the numbering of each method step is only for the purpose of identifying each method step, and not for limiting the order of each method or limiting the scope of the present invention. Changes or adjustments to their relative relationships, without substantial changes to the technical content, can also be considered as within the scope of the present invention.

[0022] The raw materials and instruments used in the examples are not subject to any specific restrictions on their source; they can be purchased from the market or prepared according to conventional methods known to those skilled in the art.

[0023] In existing technologies, Low-Rank Adaptation (LoRA) efficiently fine-tunes the weight matrix by decomposing it into low-rank matrices, thus reducing the number of parameters. However, LoRA's loss function design is complex (e.g., the singular value loss function involves freezing principal component parameters), and it does not generalize to ordinary training, resulting in significant computational overhead.

[0024] To address the aforementioned problems, according to a first aspect of the present invention, a method for training a large language model is provided, such as... Figure 1 As shown, the method may specifically include the following steps: S101, Determine the task loss based on the prediction results and true labels corresponding to the current batch of training data of the large language model; S102, determine the regularization loss based on the singular values ​​of the first low-rank matrix or the second low-rank matrix corresponding to the weight matrix of the target layer in the large language model; S103, determine the total loss of the training data in the current batch based on the task loss and regularization loss; S104, Update the parameters of the first low-rank matrix and the second low-rank matrix according to the total loss; S105, if the large language model meets the preset training termination conditions, update the target layer based on the first low-rank matrix and the second low-rank matrix.

[0025] The technical solution of this embodiment is mainly aimed at pre-trained large language models. In the LoRA fine-tuning of large language models, for a specific domain task, such as text processing tasks for agricultural texts, the large language model is fine-tuned using agricultural text training data in the agricultural domain.

[0026] In one example, in step S101, a batch of agricultural text training data is input into the large language model. The agricultural text training data consists of text in the agricultural field. The large language model then obtains prediction results for the agricultural text. The prediction results are compared with the true labels of the training text to calculate the standard task loss.

[0027] In this embodiment, in step S102, all parameters of the pre-trained large language model are frozen, and the weights of the large language model remain unchanged during LoRA fine-tuning. In this embodiment, the target layer in the large language model includes, but is not limited to, the attention model, and can also be a specified layer in the attention module.

[0028] Determine the singular values ​​of either the first or second low-rank matrix of the target layer of the large language model, and determine the regularization loss based on the singular values ​​of either the first or second low-rank matrix. In one example, approximation methods such as power iteration are used, or the Frobenius norm of the matrix is ​​used directly as an approximation of the sum of squared singular values.

[0029] Next, in step S103, the task loss and regularization loss are weighted and summed to obtain the total loss.

[0030] In step S104, the gradients of all trainable parameters (i.e., parameters in all low-rank matrices) of the total loss are calculated, and then the parameters of the first low-rank matrix and the second low-rank matrix are determined and updated.

[0031] Finally, in step S105, if the large language model meets the preset training termination condition, the target layer is updated based on the first low-rank matrix and the second low-rank matrix. If the large language model does not meet the preset training termination condition, steps S101-S104 are repeated until the large language model meets the preset training condition. The preset training termination condition includes, but is not limited to, the large language model converging or a preset number of iterations.

[0032] In this embodiment, the preset training termination conditions include, but are not limited to, the convergence of the large language model or the preset number of iterations.

[0033] In one example of this embodiment, a pre-trained large language model is trained using agricultural text training data. After meeting preset training termination conditions and updating the target layer based on the first and second low-rank matrices, the parameters of the target layer are saved. This large language model can perform tasks such as crop variety recommendation and crop irrigation plan planning on agricultural text data.

[0034] This invention determines the task loss based on the prediction results and true labels corresponding to the current batch of training data in a large language model; it determines the regularization loss based on the singular values ​​of the first or second low-rank matrix corresponding to the weight matrix of the target layer in the large language model; it determines the total loss of the training data in the current batch based on the task loss and the regularization loss; it updates the parameters of the first and second low-rank matrices based on the total loss; and it updates the target layer based on the first and second low-rank matrices when the large language model meets the preset training termination conditions. By calculating the regularization loss simultaneously with the task loss, and then determining the total loss based on the task loss and the regularization loss, the optimization process is guided to automatically and smoothly reduce the redundant complexity in the low-rank matrix while learning new tasks. This solves the problem in existing technologies where the inability to reduce model complexity leads to the model's inability to focus on the main features of a specific domain of the samples.

[0035] Optionally, in this embodiment, determining the regularization loss based on the singular values ​​of the low-rank matrix of the target layer in the large language model includes: performing singular value decomposition on the first or second low-rank matrix being trained in the current batch to obtain singular values; and performing L2 regularization on the singular values ​​to obtain the regularization loss.

[0036] Specifically, in this embodiment, the weight matrix W of each layer introduces a pair of trainable first low-rank matrices A and second low-rank matrices B. For example, the shape of the weight matrix W is... Set the rank r (for example, set...). r =8), from which we can obtain the first low-rank matrix A (shape is ). ) and the second low-rank matrix B (shape is ).

[0037] In one example, singular value decomposition (SVD) is performed on the first low-rank matrix A being trained in the current batch to obtain all its singular values. It should be noted that one of the low-rank matrices can be chosen based on practical experience (such as the first low-rank matrix A or the second low-rank matrix B). Specifically, approximation methods such as power iteration or directly using the Frobenius norm of the matrix can be used as approximations of the sum of squared singular values.

[0038] In this embodiment, the regularization term is calculated using the following formula: (1) Optionally, in this embodiment, determining the total loss of the agricultural text training data in the current batch based on the task loss and regularization loss includes: weighted summing of the task loss and regularization loss to obtain the total loss of the training data in the current batch, which can specifically be: (2) in, For the total loss, For mission losses, For singular value regularization, The regularization coefficient is . It should be noted that the strength of regularization is used to control it. If the value is too large, the model will oversuppress the singular values, which may prevent it from effectively learning the task knowledge and lead to underfitting. If it's too small, the regularization effect will be weak, close to that of the original LoRA; At a moderate level, a good balance is struck between fitting the task and maintaining the simplicity of large language models. This is used to impose constraints on the distribution of singular values ​​of the first low-rank matrix A or the second low-rank matrix B, to penalize larger singular values, thereby reducing model complexity.

[0039] The regularization loss is determined based on singular values, including:

[0040] in, These are the singular values ​​of either the first low-rank matrix A or the second low-rank matrix B.

[0041] Optionally, in this embodiment, step S4 includes, but is not limited to: determining the gradient of the parameters of the first low-rank matrix and the second low-rank matrix based on the total loss; and updating the parameters of the first low-rank matrix and the second low-rank matrix using a gradient descent algorithm.

[0042] Specifically, calculate the total loss. The gradients of all trainable parameters (i.e., all parameters in the first low-rank matrix A and the second low-rank matrix B) are calculated, and then the parameters of the first low-rank matrix A and the second low-rank matrix B are updated using a gradient descent algorithm (such as Adam). The learning rates for the first low-rank matrix and the second low-rank matrix are set to different values.

[0043] Optionally, in this embodiment, step S104 includes, but is not limited to: determining the increment matrix based on the first low-rank matrix A and the second low-rank matrix B; and determining the final weights of the target layer based on the increment matrix and the original weights of the target layer.

[0044] Specifically, repeat steps S101-S104 until the model's performance on the validation set no longer improves or reaches the preset number of training epochs. It should be noted that during forward propagation, the original W remains unchanged, but an additional increment matrix is ​​calculated. .

[0045] After training is complete, the increment matrix is ​​determined based on the first low-rank matrix A and the second low-rank matrix B, including:

[0046] in, It is an incremental matrix. Let A be the first low-rank matrix. B is the second low-rank matrix; Finally, based on the increment matrix and the original weights of the target layer, the final weights of the target layer are determined, including:

[0047] in, For the final weight, These are the original weights.

[0048] Finally, save. This allows us to obtain the final large language model after fine-tuning, which can be used for reasoning tasks in specific domains, such as text recognition in the agricultural field.

[0049] Optionally, in this embodiment, the target layer is the attention module in a large language model.

[0050] Specifically, this includes, but is not limited to, the Q (Query), K (Key), V (Value), and O (Output) projection layers in the attention mechanism of large language models.

[0051] According to a second aspect of the present invention, a large language model training apparatus is provided, such as... Figure 2 As shown, the device includes: The first determining module 21 is used to determine the task loss based on the prediction results and the true labels corresponding to the training data of the current batch of the large language model. The second determining module 22 is used to determine the regularization loss based on the singular values ​​of the first low-rank matrix or the second low-rank matrix corresponding to the weight matrix of the target layer in the large language model. The third determining module 23 is used to determine the total loss of the training data in the current batch based on the task loss and the regularization loss; The first update module 24 is used to update the parameters of the first low-rank matrix and the second low-rank matrix according to the total loss; The second update module 25 is used to update the target layer based on the first low-rank matrix and the second low-rank matrix when the large language model meets the preset training termination condition.

[0052] Optionally, in this embodiment, the second determining module 22 includes: The first processing submodule is used to perform singular value decomposition on the first low-rank matrix or the second low-rank matrix being trained in the current batch to obtain the singular values. The second processing submodule is used to perform L2 regularization on the singular values ​​to obtain the regularization loss.

[0053] Optionally, in this embodiment, the third determining module 23 includes: The third processing submodule is used to perform a weighted summation of the task loss and the regularization loss to obtain the total loss of the training data in the current batch.

[0054] Optionally, in this embodiment, the third determining module 23 includes: The first determining submodule is used to determine the gradient of the parameters of the first low-rank matrix and the second low-rank matrix based on the total loss; The first update submodule is used to update the parameters of the first low-rank matrix and the second low-rank matrix using the gradient descent algorithm.

[0055] Optionally, in this embodiment, the learning rate of the first low-rank matrix and the learning rate of the second low-rank matrix are set to different values.

[0056] Optionally, in this embodiment, the second update module 26 includes: The second determining submodule is used to determine the increment matrix based on the first low-rank matrix and the second low-rank matrix; The third determining submodule is used to determine the final weights of the target layer based on the incremental matrix and the original weights of the target layer.

[0057] Optionally, in this embodiment, determining the increment matrix based on the first low-rank matrix and the second low-rank matrix includes:

[0058] in, It is an incremental matrix. Let be the first low-rank matrix. This is the second low-rank matrix; The step of determining the final weights of the target layer based on the incremental matrix and the original weights of the target layer includes:

[0059] in, For the final weight, The original weights are given.

[0060] A third aspect of this application also provides an electronic device, such as Figure 3 As shown, it includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions executable by the at least one processor 301, the instructions being executed by the at least one processor 301 to enable the at least one processor 301 to execute the large language model training methods in the above embodiments.

[0061] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0062] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0063] A fourth aspect of this application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements an embodiment of the above-described large language model training method.

[0064] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0067] The foregoing description of specific exemplary embodiments of this application is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for training a large language model, characterized in that, The method includes: The task loss is determined based on the prediction results and true labels corresponding to the current batch of training data of the large language model. The regularization loss is determined based on the singular values ​​of the first low-rank matrix or the second low-rank matrix corresponding to the weight matrix of the target layer in the large language model. Based on the task loss and the regularization loss, determine the total loss of the training data in the current batch; Update the parameters of the first low-rank matrix and the second low-rank matrix based on the total loss; When the large language model meets the preset training termination condition, the target layer is updated according to the first low-rank matrix and the second low-rank matrix.

2. The method according to claim 1, characterized in that, Based on the singular values ​​of the low-rank matrix of the target layer in the large language model, the regularization loss is determined as follows: Perform singular value decomposition on the first low-rank matrix and / or the second low-rank matrix currently being trained in the batch to obtain the singular values; The singular values ​​are subjected to L2 regularization to obtain the regularization loss.

3. The method according to claim 2, characterized in that, Determining the total loss of the training data in the current batch based on the task loss and the regularization loss includes: performing a weighted summation of the task loss and the regularization loss to obtain the total loss of the training data in the current batch.

4. The method according to claim 1, characterized in that, The step of updating the parameters of the first low-rank matrix and the second low-rank matrix based on the total loss includes: The gradients of the parameters of the first low-rank matrix and the second low-rank matrix are determined based on the total loss. The parameters of the first low-rank matrix and the second low-rank matrix are updated using the gradient descent algorithm.

5. The method according to claim 4, characterized in that, The learning rate of the first low-rank matrix and the learning rate of the second low-rank matrix are set to different values.

6. The method according to claim 1, characterized in that, The step of updating the target layer based on the first low-rank matrix and the second low-rank matrix includes: The increment matrix is ​​determined based on the first low-rank matrix and the second low-rank matrix; The final weights of the target layer are determined based on the incremental matrix and the original weights of the target layer.

7. The method according to claim 6, characterized in that, The step of determining the increment matrix based on the first low-rank matrix and the second low-rank matrix includes: in, It is an incremental matrix. Let be the first low-rank matrix. This is the second low-rank matrix; The step of determining the final weights of the target layer based on the incremental matrix and the original weights of the target layer includes: in, For the final weight, The original weights are given.

8. A large language model training device, characterized in that, include: The first determination module is used to determine the task loss based on the prediction results and the true labels corresponding to the current batch of training data of the large language model. The second determining module is used to determine the regularization loss based on the singular values ​​of the first low-rank matrix or the second low-rank matrix corresponding to the weight matrix of the target layer in the large language model. The third determining module is used to determine the total loss of the training data in the current batch based on the task loss and the regularization loss; The first update module is used to update the parameters of the first low-rank matrix and the second low-rank matrix according to the total loss; The second update module is used to update the target layer based on the first low-rank matrix and the second low-rank matrix when the large language model meets the preset training termination condition.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the large language model training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the large language model training method as described in any one of claims 1 to 7.

Images