Model training method and device, computer device, and storage medium

Abstract

The application relates to a model training method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining a comment model, a language model and a reward model; training the reward model based on preset positive and negative samples, to obtain a trained reward model; training the comment model and the language model for multiple rounds based on the trained reward model, and inputting a preset first model instruction into the language model to be trained in the current round of the multiple-round training process, to obtain predicted reply content corresponding to the first model instruction; calling the trained reward model and the comment model to be trained in the current round to score the predicted reply content, to obtain a first score result and a second score result; and adjusting parameters of respective adapters of the comment model and the language model to be trained in the current round based on the first score result and the second score result. The method can save resources.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of science and technology, machine learning models have emerged. A machine learning model is a model that learns from data and can make predictions or decisions based on unknown data. For example, in the case of a language model, it can be used to complete dialogues with users and generate copy.

[0003] To ensure that machine learning models output more accurate results, they need to be trained before use. Currently, this primarily involves inputting training samples into the model to adjust its parameters. However, as machine learning models grow increasingly large, the number of training parameters also increases, requiring multiple graphics cards to start training and resulting in significant resource consumption. Summary of the Invention

[0004] Therefore, it is necessary to provide a model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can save resources in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a model training method, the method comprising:

[0006] Obtain a comment model, a language model, and a reward model; the comment model, the language model, and the reward model each include a base model and an adapter with fewer parameters than the base model;

[0007] The reward model is trained based on a preset pair of positive and negative samples to obtain the trained reward model.

[0008] The comment model and the language model are trained in multiple rounds based on the trained reward model. In the current round of the training process, a preset first model instruction is input into the language model to be trained in the current round to obtain the predicted response content corresponding to the first model instruction.

[0009] The trained reward model and the comment model to be trained in the current round are used to score the predicted response content to obtain a first score result and a second score result.

[0010] Based on the first and second scoring results, the parameters of the adapters for the comment model and language model to be trained in the current round are adjusted respectively to obtain the comment model and language model to be trained in the next round.

[0011] Secondly, this application also provides a model training apparatus, the apparatus comprising:

[0012] The model acquisition module is used to acquire a comment model, a language model, and a reward model; the comment model, the language model, and the reward model each include a base model and an adapter with fewer parameters than the base model.

[0013] The reward model training module is used to train the reward model based on preset positive and negative sample pairs to obtain the trained reward model.

[0014] The joint training module is used to train the comment model and the language model in multiple rounds based on the trained reward model. In the current round of the multi-round training process, a preset first model instruction is input into the language model to be trained in the current round to obtain the predicted response content corresponding to the first model instruction. The trained reward model and the comment model to be trained in the current round are used to score the predicted response content to obtain a first score result and a second score result. Based on the first score result and the second score result, the parameters of the adapters of the comment model and the language model to be trained in the current round are adjusted to obtain the comment model and the language model to be trained in the next round.

[0015] In one embodiment, the model acquisition module is further configured to obtain a base model, an adapter, and a value head; the value head refers to a machine learning module used to evaluate the value of the current state; a language model is generated based on the base model and the adapter; and a reward model and a comment model are generated based on the base model, the adapter, and the value head.

[0016] In one embodiment, the model acquisition module is further configured to acquire the weight matrix of the base model and determine the dimension of the weight matrix; generate a low-rank matrix based on the dimension of the weight matrix; wherein the dimension of the low-rank matrix is ​​less than the dimension of the weight matrix; and generate an adapter based on the low-rank matrix.

[0017] In one embodiment, the reward model training module is further configured to acquire positive and negative sample pairs, the positive and negative sample pairs including positive samples and negative samples; the positive samples include a second model instruction and high-quality response content generated based on the second model instruction that meets the high-quality response conditions; the negative samples include the second model instruction and low-quality response content generated based on the second model instruction that does not meet the high-quality response conditions; the positive samples are input into the reward model, and the reward model is invoked to score the high-quality response content based on the second model instruction to obtain positive sample scoring results; the negative samples are input into the reward model, and the reward model is invoked to score the low-quality response content based on the second model instruction to obtain negative sample scoring results; the parameters of the reward model are adjusted according to the positive sample scoring results and the negative sample scoring results to obtain a trained reward model.

[0018] In one embodiment, the reward model includes a base model, a value head, and an adapter; the reward model training module is further configured to determine the score difference between the positive sample rating results and the negative sample rating results; fix the parameters of the base model in the reward model, and adjust the parameters of the value head and adapter in the reward model in the direction of increasing the score difference to obtain the trained reward model.

[0019] In one embodiment, the joint training module is further configured to synchronize the parameters of the adapter of the trained reward model to the adapter of the comment model to obtain the comment model to be trained in the first round; obtain a preset value and set the parameters of the adapter in the language model to the preset value to obtain the language model to be trained in the first round.

[0020] In one embodiment, the joint training module is further configured to determine the comment model loss based on the first rating result and the second rating result, and adjust the parameters of the adapter of the comment model to be trained in the current round based on the comment model loss; and to determine the language model loss based on the predicted response content, the first rating result and the second rating result, and adjust the parameters of the adapter of the language model to be trained in the current round based on the language model loss.

[0021] In one embodiment, the first model instruction is located in a first model instruction set; in another embodiment, the joint training module is further configured to, when the language model to be trained in the current round outputs multiple predicted response contents based on multiple first model instructions in the model instruction set, call the trained reward model and the comment model to be trained in the current round to score each predicted response respectively, to obtain multiple first scoring results and multiple second scoring results; determine the mean absolute error between the multiple first scoring results and the multiple second scoring results; and determine the comment model loss based on the mean absolute error.

[0022] In one embodiment, the joint training module is further configured to determine an overall response score based on the first scoring result and the second scoring result; determine a penalty term loss based on the difference between the predicted response content and the response content output by the base model based on the instructions of the first model; and determine a language model loss based on the overall response score and the penalty term loss.

[0023] In one embodiment, the joint training module is further configured to obtain the first logarithmic probability corresponding to each word in the predicted response content output by the language model to be trained in the current round; concatenate the first model instruction and the predicted response content to obtain a concatenation instruction; input the concatenation instruction to the base model, call the base model to output the second logarithmic probability corresponding to each word in the predicted response content; and determine the penalty term loss based on the first logarithmic probability and the second logarithmic probability.

[0024] In one embodiment, the joint training module is further configured to obtain the first score result output by the trained reward model in multiple consecutive training rounds; when the difference between the obtained first score results is less than or equal to a preset difference threshold, the training of the language model is stopped to obtain the language model after multiple training rounds.

[0025] In one embodiment, the model training device further includes using a model to display icons corresponding to the language model after multiple rounds of training; in response to a trigger operation on the icons, displaying a content display interface; the content display interface includes a model instruction input box and a response display area; in response to a model instruction input operation, inputting a third model instruction through the model instruction input box into the language model after multiple rounds of training, calling the language model after multiple rounds of training to output the response content corresponding to the third model instruction; and displaying the response content corresponding to the third model instruction in the response display area.

[0026] Thirdly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in any of the model training methods provided in the embodiments of this application.

[0027] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps in any of the model training methods provided in the embodiments of this application.

[0028] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the model training methods provided in the embodiments of this application:

[0029] The aforementioned model training method, apparatus, computer equipment, storage medium, and computer program product, by acquiring a comment model, a language model, and a reward model, can train the reward model to obtain a trained reward model. Using the trained reward model, the comment model and language model can be trained multiple times. In the current round, the predicted response content is output by the language model to be trained in the current round, and the predicted response content is scored using the trained reward model and the comment model to be trained in the current round, obtaining a first score result and a second score result. Based on the first and second score results, the parameters of the adapters for the comment model and language model to be trained in the current round can be adjusted to obtain the next round of trained comment and language models. Since only the adapters with fewer parameters than the base model need to be adjusted, compared to traditionally adjusting all parameters in a machine learning model, this application can significantly reduce the number of parameters that need to be adjusted, thus enabling model training without multiple graphics cards, reducing resource consumption, and saving computer resources.

[0030] Furthermore, since the comment model, language model, and reward model share the parameters of the base model, multiple models can be obtained by loading only one base model during training, which greatly reduces the consumption of GPU memory and further saves computer resources. Attached Figure Description

[0031] Figure 1 This is a diagram illustrating the application environment of a model training method in one embodiment;

[0032] Figure 2 This is a flowchart illustrating a model training method in one embodiment;

[0033] Figure 3 This is a schematic diagram of positive and negative sample pairs in one embodiment;

[0034] Figure 4 This is a schematic diagram illustrating the use of a reward model in one embodiment;

[0035] Figure 5 This is a schematic diagram of iterative training in one embodiment;

[0036] Figure 6 This is a schematic diagram illustrating the current training round of the comment model and the language model in one embodiment;

[0037] Figure 7 This is a schematic diagram illustrating the generation of a comment model, a language model, and a reward model based on a base model in one embodiment.

[0038] Figure 8 This is a flowchart illustrating the training process of a reward model in one embodiment.

[0039] Figure 9 This is a schematic diagram illustrating the training of the comment model and the language model in one embodiment;

[0040] Figure 10 This is a schematic diagram illustrating the determination of language model loss in one embodiment;

[0041] Figure 11 This is a schematic diagram of the content display interface in one embodiment;

[0042] Figure 12 This is a schematic diagram of the overall process of model training in one embodiment;

[0043] Figure 13 This is a schematic diagram of the overall model training process in another embodiment;

[0044] Figure 14 This is a structural block diagram of a model training device in one embodiment;

[0045] Figure 15 This is an internal structural diagram of a computer device in one embodiment;

[0046] Figure 16 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] The model training method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on the cloud or other servers. Both terminal 102 and server 104 can be used independently to execute the model training method provided in this embodiment. Terminal 102 and server 104 can also be used collaboratively to execute the model training method provided in this embodiment. Taking the example of terminal 102 and server 104 collaboratively executing the model training method provided in this embodiment, when a user clicks the training start icon displayed on terminal 102, terminal 102 responds to the click operation by generating a model training start command and sending it to server 104. When server 104 receives the model training start command, it can start model training to obtain a trained language model.

[0049] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers. The server 104 can also be a cloud server.

[0050] It should be noted that the terms "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Unless the context clearly indicates otherwise, the singular forms "a," "one," or "the," etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. The quantities of "multiple" or "multiple copies" mentioned in the embodiments of this application all refer to a quantity of "at least two," for example, "multiple" means "at least two," and "multiple copies" means "at least two copies."

[0051] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows:

[0052] Reinforcement learning (RL) is a field within machine learning that emphasizes how to act based on the environment to maximize expected benefits. It is the third fundamental machine learning method, alongside supervised and unsupervised learning. Unlike supervised learning, reinforcement learning does not require labeled input-output pairs, nor does it require precise correction of suboptimal solutions. Its focus is on finding a balance between exploration (of the unknown) and exploitation (of existing knowledge)—the "exploration-exploitation" exchange in reinforcement learning.

[0053] Reinforcement Learning From Human Feedback (RLHF) is a machine learning technique that directly trains a reward model based on human feedback. This model serves as the reward function in reinforcement learning, and algorithms such as proximal policy optimization are used to optimize the agent's policy. The reward model is pre-trained before policy optimization to predict whether a given output is good (high reward) or bad (low reward). RLHF can improve the robustness and exploratory nature of reinforcement learning agents, especially suitable for situations where the reward function is sparse or noisy (uncertain). Training with RLHF can further enhance the ability of large models to follow human instructions, enabling them to learn behavioral patterns that better align with human expectations and values.

[0054] The model training method in this application is a reinforcement learning model training method based on human feedback. This method involves the interaction of a language model, a comment model, a reward model, and a base model. The goal of the entire training process is to make the content generated by the language model get higher and higher scores, while keeping the difference in output distribution between the language model and the base model within a certain limit. This allows for fine-tuning of the output results to better follow human input instructions.

[0055] A language model is a probability distribution model of words in a natural language. It provides a sequence of words and calculates their probability of occurrence. It can also predict the next most likely word. It has many applications in the field of natural language processing.

[0056] The base model is an artificial neural network large language model, typically with billions to hundreds of billions of weight parameters. The base model in this application has been fine-tuned on a large number of samples. Through thorough fine-tuning, the base model can acquire rich world knowledge and powerful language understanding capabilities, demonstrating excellent performance in specific business scenarios.

[0057] Large Language Models (LLMs) are artificial neural network language models that typically have billions to hundreds of billions of weight parameters. They are pre-trained on large amounts of unlabeled text using self-supervised or semi-supervised learning. The pre-training task is to predict the next word given the preceding context. Through pre-training, large language models acquire rich world knowledge and powerful language understanding capabilities, demonstrating excellent performance in most scenarios.

[0058] Fine-tuning training: This involves fine-tuning the model on a specific task to adapt to the needs of different downstream tasks, such as text classification, named entity recognition, and image classification. This approach can significantly improve the model's performance on specific tasks. For language models, fine-tuning is often done using general dialogue commands to train a model that can follow human instructions and engage in dialogue with humans.

[0059] Comment model: This is a scoring model that typically uses a trained reward model to initialize parameters and continuously updates the parameters during subsequent training to improve the scoring criteria.

[0060] Reward model: This is a pre-scoring model that is trained to score the content output by the language model more accurately.

[0061] The adapter is a machine learning module composed of low-rank matrices. After adding the adapter to the original matrix, during model training, the original model parameters are frozen, and only the adapter parameters are updated. It can achieve near-full-parameter model fine-tuning performance with less GPU memory requirements. Language models with billions of parameters only need to be tuned for a few million.

[0062] In one embodiment, such as Figure 2 As shown, a model training method is provided. The method is illustrated using a computer device as an example. The computer device can be, for example, a... Figure 1 The terminal or server in the system. The model training method includes the following steps:

[0063] Step 202: Obtain the comment model, language model, and reward model; the comment model, language model, and reward model each include a base model and an adapter with fewer parameters than the base model.

[0064] Specifically, when training a reinforcement learning model based on human feedback is required, the computer device can acquire the comment model, language model, and reward model involved in the training process. Each of these models includes a base model and an adapter, with the adapter having fewer parameters than the base model.

[0065] In one embodiment, the comment model, language model, and reward model can share a common base model. Therefore, the computer device only needs to load one base model and add corresponding adapters to each to obtain the comment model, language model, and reward model. Because the comment model, language model, and reward model share a common base model, the computer device only needs to load one base model to obtain the comment model, language model, and reward model, without having to load their respective base models separately, thus significantly saving the computer device's memory resources.

[0066] In one embodiment, the base model may be a large language model that has been finely trained; for example, the base model may specifically be a Transformer model.

[0067] In one embodiment, for a basic model, the computer device can create a model instance and load a parameter file using a preset parameter loading function. Then, using a preset parameter loading method, the parameters in the parameter file are loaded into the model instance, thus realizing the loading of the basic model.

[0068] In one embodiment, not only can the parameters of the base models in the comment model, language model, and reward model be shared, but the parameters of the adapters in the initially acquired comment model, language model, and reward model can also be shared, thereby further saving the memory resources of the computer device.

[0069] Step 204: Train the reward model based on the preset positive and negative sample pairs to obtain the trained reward model.

[0070] Specifically, the computer device can acquire positive and negative sample pairs. A positive and negative sample pair refers to a pair of samples that includes a positive sample and a negative sample. A positive sample is a sample that conforms to the target concept or has specific characteristics; for example, a positive sample could be a sample that conforms to human preferences. A negative sample is a sample that does not conform to the target concept or has specific characteristics; for example, a negative sample could be a sample that does not conform to human preferences. Furthermore, after acquiring the positive and negative sample pairs, the computer device can train the reward model based on the positive and negative sample pairs to adjust the parameters of the reward model and obtain a trained reward model.

[0071] In one embodiment, model instructions for generating positive and negative sample pairs can be obtained. These model instructions refer to model prompts. Model prompts are a technique used when interacting with large models to guide the model to generate the expected output in an explicit and organized manner. For ease of description, the model instructions used to generate positive and negative sample pairs will be referred to as second model instructions below. The second model instructions are input into the base model, which outputs multiple responses. The superiority / minority relationships among the multiple responses are compared, and positive and negative sample pairs are generated based on these relationships and the multiple responses.

[0072] In one embodiment, the second model instruction can specifically be a question. After the question is input into the base model, the base model can output multiple different responses to the question. The quality of the different responses can be compared, and positive samples are generated based on the better responses, while negative samples are generated based on the worse responses. By obtaining positive and negative samples, positive and negative sample pairs are obtained. For example, a computer device can display the question and each response, allowing users to rate each response according to their preferences; the higher the score, the better.

[0073] For example, refer to Figure 3 , Figure 3 The diagram illustrates a pair of positive and negative samples in one embodiment. When the question is "Convert 0.84545 to a percentage," the response in a positive sample could be "0.84545 is 84.545%", and the response in a negative sample could be "0.84545 can be converted to 84.545 / 100". When the question is "Generate a short poem based on the word 'moonlight'", the response in a positive sample could be "The moon is bright, the stars are twinkling, the soft light illuminates the dark corners; time flies, years pass quickly, may beauty remain forever in the world," and the response in a negative sample could be "The bright moonlight shines on the earth, shines on the quiet night, the faint light illuminates our hearts, allowing us to feel the beauty of life."

[0074] In one embodiment, the response output by the base model may include a reasoning process and a final conclusion regarding the problem. For example, the second model instruction may be "What are the application scenarios of the worst-case principle?", and the response output by the base model based on the second model instruction may be "Since the worst-case principle can analyze and solve problems from the perspective of the worst-case scenario... therefore, the worst-case principle can be used for ball-drawing problems, seating problems, and card-drawing problems." Here, "Since the worst-case principle can analyze and solve problems from the perspective of the worst-case scenario..." in the response output based on the second model instruction represents the reasoning process, and "the worst-case principle can be used for ball-drawing problems, seating problems, and card-drawing problems" represents the final conclusion. Labels may be set for the second model instruction; for example, the label may be "The worst-case principle can be used for ball-drawing problems, seating problems, card-drawing problems, and sock color sorting problems." The computer device can evaluate the correctness of each response based on the labels to obtain the correctness evaluation results for each response. For example, the more similar the final conclusion is to the label, the higher the correctness score of the response. Computer equipment can determine the logical coherence assessment result of a response based on the logical coherence of the reasoning process within the response. For example, the computer equipment can determine whether the reasoning process is logically sound and consistent with its inconsistencies. If it is logically sound and consistent with its inconsistencies, the logical coherence assessment score is higher; if it is logically sound or inconsistent with its inconsistencies, the logical coherence assessment score is lower. Furthermore, the computer equipment determines the quality relationship between responses based on their individual correctness assessment results and logical coherence assessment results. For example, it adds the correctness score and the logical coherence assessment score of each response to obtain a total score; the higher the total score, the better the response.

[0075] In one embodiment, by using positive and negative samples to train the reward model, it is possible to correctly score the response content based on the output of the second model's instructions. For example, refer to Figure 4 The structure of a reward model generally consists of an adapter and a value head following a base model. The value head can be a multi-layer perceptron (MLP). This MLP can compress a high-dimensional vector into a single score output. For example, the second model instruction (i.e., the question) "convert 0.84545 into a percentage" and the corresponding response "the percentage of 0.84545 is 84.545%" can be input into the reward model, and the reward model can then output a score of "0.315" for "the percentage of 0.84545 is 84.545%". Figure 4 A schematic diagram illustrating the use of a reward model in one embodiment is shown.

[0076] Step 206: Based on the trained reward model, the comment model and language model are trained in multiple rounds. In the current round of the multi-round training process, the preset first model instruction is input into the language model to be trained in the current round to obtain the predicted response content corresponding to the first model instruction.

[0077] Specifically, after training the reward model, the comment model and language model can be trained multiple times based on the trained reward model. This multi-round training continuously improves the scoring criteria of the comment model, and consequently, the accuracy of the language model's output is continuously improved based on the increasingly higher scoring criteria. For example, refer to... Figure 5 The goal of training is to continuously improve the score of the language model's response results, realizing the link of human feedback to optimize model generation. The entire iterative process is as follows: after training the comment model and reward model to be trained in the current round based on the trained reward model, the comment model and reward model to be trained in the next round can be obtained. Thus, when entering the next round of training, the comment model and reward model to be trained in the next round can be used as the new comment model and reward model to be trained in the current round. This iterative process continues until the preset stopping condition is met. Figure 5 A schematic diagram of iterative training in one embodiment is shown.

[0078] Due to the improvement in scoring standards, for the same response output by the language model, the scoring standards of the comment model are continuously improved during training because the parameters of the comment model are adjusted. As a result, the score of the comment model for the same response obtained in the current training round will be lower than the score obtained in the previous training round. Therefore, in the current training round, the language model will realize that its output response does not yet meet the scoring standards of the comment model based on the lower score, and will adjust its parameters in the direction of meeting the scoring standards. In this way, after multiple rounds of training, the language model will be able to output more accurate response content.

[0079] To better understand this application, the current training round in a multi-round training process is described in detail below. In the current training round of a multi-round training process for the comment model and the language model, the computer device can acquire model instructions. For ease of description, the model instructions used to train the comment model and the language model will be referred to as the first model instructions. The computer device can input the first model instructions into the language model to be trained in the current round, and the language model to be trained in the current round will output response content. For ease of description, the response content output by the language model during training will be referred to as the predicted response content.

[0080] It is easy to understand that, for the language model to be trained in the current round of training, when the current round is the first round, the language model to be trained in the current round can be the language model obtained in step 202; when the current round is a round after the first round, the language model to be trained in the current round can be the language model obtained in the previous round of training.

[0081] Step 208: Call the trained reward model and the comment model to be trained in the current round to score the predicted response content, and obtain the first score result and the second score result.

[0082] Specifically, in the current training round, after obtaining the predicted content output by the language model to be trained in the current round, the predicted content can be input into the trained reward model to score the predicted content and obtain a first score result. Additionally, the computer device can input the predicted content into the comment model to be trained in the current round to score the predicted content and obtain a second score result.

[0083] It is easy to understand that, for the comment model to be trained in the current round of training, when the current round is the first round, the comment model to be trained in the current round can be the model after initializing the comment model obtained in step 202 based on the trained reward model; when the current round is a round after the first round, the comment model to be trained in the current round can be the comment model obtained in the previous round.

[0084] Step 210: Based on the first and second scoring results, adjust the parameters of the adapters for the comment model and language model to be trained in the current round to obtain the comment model and language model to be trained in the next round.

[0085] Specifically, after obtaining the first and second rating results, the computer device can generate a loss for training the comment model to be trained in the current round based on the first and second rating results. For example, the first and second rating results can be input into a preset comment model loss calculation function, and the preset comment model loss calculation function can output the loss used to train the comment model to be trained in the current round. Similarly, the computer device can generate a loss for training the language model to be trained in the current round based on the first and second rating results. For example, the first and second rating results can be input into a preset language model loss calculation function, and the preset language model loss calculation function can output the loss used to train the language model to be trained in the current round.

[0086] Here, loss is a numerical value calculated using a specific loss function. Essentially, it quantifies the degree of mismatch between the model's predicted value and the true value. For ease of description, the loss used to train the comment model in the current round will be called the comment model loss, and the loss used to train the language model in the current round will be called the language model loss.

[0087] Furthermore, after obtaining the comment model loss, the computer device can adjust the parameters of the adapter for the comment model to be trained in the current round based on the comment model loss. For example, the parameters of the adapter for the comment model to be trained in the current round can be adjusted by minimizing the comment model loss to obtain the comment model to be trained in the next round. Similarly, after obtaining the language model loss, the computer device can adjust the parameters of the adapter for the language model to be trained in the current round based on the language model loss. For example, the parameters of the adapter for the language model to be trained in the current round can also be adjusted by minimizing the language model loss to obtain the language model to be trained in the next round.

[0088] In one embodiment, reference Figure 6 , Figure 6 This diagram illustrates the current training round of the comment model and language model in one embodiment. First, given the input of a first model instruction, the language model generates a corresponding predicted response. The reward model and comment model score the responses based on the first model instruction and the predicted responses, obtaining a first score and a second score. These scores are then used to adjust the parameters of the adapters for the comment model and language model. The predicted responses can also be incorporated when adjusting the language model's adapter parameters. The purpose of training the comment model is to ensure that its second comment result continuously fits the reward model's score, thus achieving an awareness of the current capabilities of the language model and preventing it from reaching a bottleneck. The language model loss is generated based on the first and second score results and the predicted responses, and the policy gradient is used to update the parameters of the language model's adapter, continuously optimizing towards higher scores.

[0089] In the above model training method, by acquiring the comment model, language model, and reward model, the reward model can be trained to obtain the trained reward model. Based on the trained reward model, the comment model and language model can be trained multiple times. In the current round, the predicted response content is output by the language model to be trained in the current round, and the predicted response content is scored by the trained reward model and the comment model to be trained in the current round, obtaining a first score result and a second score result. Based on the first and second score results, the parameters of the adapters for the comment model and language model to be trained in the current round can be adjusted respectively to obtain the next round of trained comment model and language model. Since only the parameters of the adapters, which have fewer parameters than the base model, need to be adjusted, compared to the traditional adjustment of all parameters in the machine learning model, this application can greatly reduce the number of parameters that need to be adjusted, thus enabling model training without multiple graphics cards, reducing resource consumption, and saving computer resources.

[0090] Furthermore, since the comment model, language model, and reward model share the parameters of the base model, multiple models can be obtained by loading only one base model during training, which greatly reduces the consumption of GPU memory and further saves computer resources.

[0091] In one embodiment, obtaining a comment model, a language model, and a reward model includes: obtaining a base model, an adapter, and a value head; the value head refers to a machine learning module used to evaluate the value of the current state; generating a language model based on the base model and the adapter; and generating a reward model and a comment model based on the base model, the adapter, and the value head.

[0092] Specifically, when model training is required, the computer device can first acquire a base model and an adapter, and then add an adapter after the base model to obtain a language model. For example, an adapter can be added after the last hidden layer of the base model to obtain the language model. Further, the computer device can acquire a value head, and add an adapter and a value head after the base model to obtain a reward model; for example, an adapter can be added after the last hidden layer of the base model, and a value head can be added after the adapter. The evaluation model can be consistent with the reward model; therefore, the computer device can again add an adapter and a value head after the base model to obtain the evaluation model. Here, the value head refers to the machine learning module used to evaluate the value of the current state. In the context of reinforcement learning, the state value represents the expected cumulative reward that can be obtained by following the current policy starting from that state.

[0093] In one embodiment, the value head may specifically be an MLP (Multi-Layer Perceptron) layer.

[0094] In one embodiment, reference Figure 7 , Figure 7 The illustration shows a schematic diagram of generating a comment model, a language model, and a reward model based on a base model in one embodiment. A Supervised Fine-Tuning (SFT) model can be used as the base model. An adapter is added after the base model to obtain the language model; an adapter and a value head are added after the base model to obtain the reward model; and an adapter and a value head are added again after the base model to obtain the comment model.

[0095] In the above embodiments, multiple models share the same base model, which only needs to be loaded once, thereby saving memory resources and reducing memory reuse. Furthermore, by adding an adapter to the base model, subsequent model training only requires adjusting a small number of additional fine-tuning parameters in the adapter, reducing the number of parameter adjustments. This not only improves model training efficiency but also saves the computing resources consumed when adjusting a large number of parameters.

[0096] In one embodiment, the method further includes: obtaining the weight matrix of the base model and determining the dimension of the weight matrix; generating a low-rank matrix based on the dimension of the weight matrix; wherein the dimension of the low-rank matrix is ​​less than that of the weight matrix; and generating an adapter based on the low-rank matrix.

[0097] Specifically, computer devices can generate adapters with fewer parameters based on a base model. For example, a computer device can obtain the weight matrix of the base model and use it to obtain an adapter. Here, parameters are all the adjustable parts of the machine learning model, including weights, offsets, etc. Weights are one type of parameter in a machine learning model; they are specific numerical values ​​in the model used to represent the importance of different features. During training, weights are continuously adjusted to ensure the model can better fit the data. The weight matrix is ​​a matrix composed of the weights in the base model. For example, the weight matrix W can be a d×d dimensional matrix, thus representing the weights in the base model. .

[0098] In one embodiment, the update of the weight matrix is ​​approximated by introducing a low-rank matrix. For a weight matrix W in the base model, the computer device can determine the dimension of the weight matrix and generate at least one low-rank matrix with a dimension less than that of the weight matrix. For example, when the weight matrix W is a d×d dimensional matrix, two low-rank matrices A and B can be generated with dimensions d×r and r×d respectively (where r is the rank of the low-rank matrix). These two low-rank matrices can be used as adapters in language and comment models. During model training, the update of the weight matrix is ​​approximated as W′ = W + αAB, where α is a scaling factor. By training, the parameters of A and B are learned, allowing W′ to adapt to new tasks. The final AB can be seen as a low-rank approximation of the original weight matrix W.

[0099] In the above embodiments, by generating a low-rank matrix with fewer dimensions than the base model, the number of parameters in the low-rank matrix can be less than the number of parameters in the weight matrix, thereby reducing the number of parameters that need to be adjusted during model training. For example, during model training, only the parameters of the adapters for the language model and the comment model need to be adjusted, and the number of parameters in the language model and the comment model accounts for about 0.2% of the total number of parameters, allowing a single graphics card to complete the model training, greatly reducing resource consumption.

[0100] In one embodiment, training a reward model based on preset positive and negative sample pairs to obtain a trained reward model includes: acquiring positive and negative sample pairs, each pair comprising positive samples and negative samples; positive samples comprising second model instructions and high-quality response content generated based on the second model instructions that meets the high-quality response criteria; negative samples comprising second model instructions and low-quality response content generated based on the second model instructions that does not meet the high-quality response criteria; inputting positive samples into the reward model, calling the reward model to score the high-quality response content based on the second model instructions to obtain positive sample scoring results; inputting negative samples into the reward model, calling the reward model to score the low-quality response content based on the second model instructions to obtain negative sample scoring results; and adjusting the parameters of the reward model based on the positive and negative sample scoring results to obtain a trained reward model.

[0101] Specifically, after the reward model is generated, the computer device can train the reward model. When training the reward model, the computer device can acquire positive and negative sample pairs, where each positive and negative sample pair includes positive samples and negative samples. Positive samples may include second model instructions and high-quality response content generated based on the second model instructions that meets the conditions for high-quality responses. For example, positive samples may include response content that conforms to human preferences. Negative samples may include second model instructions and low-quality response content generated based on the second model instructions that does not meet the conditions for high-quality responses. For example, negative samples may include response content that does not conform to human preferences.

[0102] Furthermore, the computer device can input positive samples into the reward model. The reward model can extract text features from the responses of positive samples, such as keywords, sentiment words, and parts of speech. Based on the extracted text features, it determines the relevance between the response and the instructions of the second model, as well as the accuracy, completeness, and fluency of the response. The reward model comprehensively considers relevance, accuracy, completeness, and fluency to score the responses in the positive samples, obtaining a positive sample score. Conversely, the computer device can input negative samples into the reward model, which scores the low-quality responses in the negative samples, obtaining a negative sample score.

[0103] Furthermore, the reward model can be trained based on positive and negative sample rating results. For example, the parameters of the reward model can be adjusted based on the difference between positive and negative sample rating results to obtain the trained reward model.

[0104] In one embodiment, the response content is determined to meet the high-quality response criteria when it satisfies human preferences; or, the response content is determined to meet the high-quality response criteria when it is strongly correlated with the input model instructions and is accurate, complete, and fluent.

[0105] In the above embodiments, the reward model is trained using positive and negative samples, enabling the trained reward model to correctly identify responses that meet the quantitative response conditions. Consequently, the comment model trained based on the trained reward model can output more accurate scores, and the language model trained based on the trained reward model can output responses that meet the quantitative response conditions.

[0106] In one embodiment, the reward model includes a base model, a value head, and an adapter; adjusting the parameters of the reward model based on positive and negative sample ratings to obtain a trained reward model includes: determining the score difference between positive and negative sample ratings; fixing the parameters of the base model in the reward model, and adjusting the parameters of the value head and adapter in the reward model in a direction that increases the score difference to obtain a trained reward model.

[0107] Specifically, both positive and negative sample ratings can be represented as a single score. After obtaining the positive and negative sample ratings, the negative sample rating can be subtracted from the positive sample rating to obtain the score difference. The computer device can fix the parameters of the base model in the reward model and adjust the parameters of the value head and adapter in the direction of increasing the score difference. For example, the computer device can obtain the reward loss Lr using the following formula and adjust the parameters of the value head and adapter in the reward model using the reward loss:

[0108]

[0109] in, For positive sample scoring results, The positive sample rating is significantly higher than the negative sample rating. The reward loss will be smaller when the positive sample rating is significantly higher than the negative sample rating, indicating that the reward model has made the correct distinction. Conversely, when the positive and negative sample ratings are similar, the reward loss will be larger, indicating that the reward model has not made the correct distinction.

[0110] In this embodiment, by adjusting the parameters of the reward model in the direction of increasing score differences, the trained reward model can distinguish responses that meet the quantitative response conditions. Furthermore, during the training of the reward model, only the parameters of the adapter and value head need to be adjusted, allowing the training process to be completed with only a small number of parameter updates. This improves the training speed, reduces training costs, and lowers the requirements for resources such as GPUs, memory, and graphics cards.

[0111] In one embodiment, reference Figure 8 , Figure 8 A flowchart illustrating the training process of a reward model in one embodiment is shown. The reward model includes an adapter with a base model appended to the last hidden layer, and a value head appended to the adapter. A set of dialogues (including positive and negative samples) can be input into the reward model, which can then output positive and negative sample ratings. A computer device can input the positive and negative sample ratings into a preset loss function to obtain a reward loss, which is used to adjust the parameters of the value head of the adapter. After obtaining the trained reward model, when using it to train a comment model and a language model, the scores output by the trained reward model can be regularized to ensure that the scores are distributed within a certain range.

[0112] In one embodiment, before the first round of training in the multi-round training process, the method further includes: synchronizing the parameters of the adapter of the trained reward model to the adapter of the comment model to obtain the comment model to be trained in the first round; obtaining preset values ​​and setting the parameters of the adapter in the language model to the preset values ​​to obtain the language model to be trained in the first round.

[0113] Specifically, after obtaining the trained reward model, the computer device can perform multiple rounds of training on the comment model and language model based on the trained reward model. Before the first round of training in the multi-round training process, the computer device can initialize the comment model and language model to obtain the comment model and language model to be trained in the first round. For the comment model, the computer device can synchronize the parameters of the adapter of the trained reward model to the adapter of the comment model to obtain the initialized comment model, and use the initialized comment model as the comment model to be trained in the first round. For example, the computer device can set the parameters of the adapter in the comment model to be consistent with the parameters of the adapter in the trained reward model, so as to synchronize the parameters of the adapter of the trained reward model to the adapter of the comment model to obtain the comment model to be trained in the first round. For the language model, the computer device can obtain preset values, such as 0, and set the parameters of the adapter in the language model to the preset values, such as setting the parameters of the adapter in the language model to 0, thereby obtaining the initialized language model, and use the initialized language model as the language model to be trained in the first round.

[0114] In this embodiment, by setting the parameters of the adapter in the comment model to be consistent with the parameters of the adapter in the trained reward model, the initialized comment model can achieve relatively accurate scoring. Further training based on this foundation can significantly shorten the training cycle and improve training efficiency. Setting the parameters of the adapter in the language model to preset values ​​facilitates subsequent adjustments to the adapter parameters, further improving training efficiency.

[0115] In one embodiment, the parameters of the adapters for the comment model and the language model to be trained in the current round are adjusted based on the first rating result and the second rating result, including: determining the comment model loss based on the first rating result and the second rating result, and adjusting the parameters of the adapter for the comment model to be trained in the current round based on the comment model loss; determining the language model loss based on the predicted response content, the first rating result and the second rating result, and adjusting the parameters of the adapter for the language model to be trained in the current round based on the language model loss.

[0116] Specifically, for the current round in a multi-round training process, after obtaining the predicted response content, the first rating result, and the second rating result, the computer device can input the first rating result and the second rating result into a preset comment model loss function, and output the loss through the comment model loss function. For ease of description, the loss used to train the comment model will be referred to as the comment model loss below. Further, the computer device can fix the parameters of the base model in the comment model to be trained in the current round, and adjust the parameters of the adapter and value head in the comment model to be trained in the current round based on the comment model loss. For example, the computer device can adjust the parameters of the adapter and value head in the comment model to be trained in the current round in the direction of minimizing the comment model loss to obtain the comment model to be trained in the next round. Correspondingly, the computer device can input the predicted response content, the first rating result, and the second rating result into a preset language model loss function, and output the loss through the language model loss function. For ease of description, the loss used to train the language model will be referred to as the language model loss below. Furthermore, the computer device can fix the parameters of the base model in the language model to be trained in the current round, adjust the parameters of the adapter in the language model to be trained in the current round based on the language model loss, and obtain the language model to be trained in the next round.

[0117] In one implementation, reference Figure 9 In the current training round, when the input content is fed into the language model to be trained in the current round, the predicted response content can be output through the base model and adapter in the language model to be trained in the current round. When the predicted response content is fed into the comment model to be trained in the current round, the second rating result can be output through the base model and adapter in the comment model to be trained in the current round. Subsequently, the model training for the current round can be performed based on the predicted response content and the second rating result. During model training, the parameters of the adapters in the language model and comment model to be trained in the current round are adjusted, while the parameters of the reward model are frozen. Since the parameter dimension of the adapters in the comment model and language model can both be d×r, which is smaller than the dimension d×d of the weight matrix of the base model, the number of training parameters is only multiple adapters, accounting for about 0.2% of the total parameters. A single 40GB graphics card is sufficient to complete the training. Furthermore, since the base model parameters of the four models are shared, only one base model parameter needs to be loaded, reducing the required video memory to less than 14GB. Figure 9 A schematic diagram illustrating the training of the comment model and the language model in one embodiment is shown.

[0118] In one embodiment, the first model instructions are located in the first model instruction set; calling the trained reward model and the comment model to be trained in the current round to score the predicted response content respectively, and obtaining a first score result and a second score result, includes: when the language model to be trained in the current round outputs multiple predicted response contents based on multiple first model instructions in the model instruction set, calling the trained reward model and the comment model to be trained in the current round to score each predicted response respectively, and obtaining multiple first score results and multiple second score results; determining the comment model loss based on the first score results and the second score results, including: determining the mean absolute error between the multiple first score results and the multiple second score results; and determining the comment model loss based on the mean absolute error.

[0119] Specifically, the parameters of the comment model can be updated using the mean absolute error between the second rating result of the comment model and the first rating result of the reward model as the training loss. More specifically, the language model can generate a batch of predicted responses for a batch of instructions. For each predicted response, the trained reward model and the comment model to be trained in the current round can be called to score, obtaining the corresponding first and second rating results. After obtaining multiple first and second rating results, the mean absolute error between the multiple first rating results and the multiple second rating results can be calculated. The parameters of the adapter and value head of the comment model can be adjusted in the direction of reducing the mean absolute error.

[0120] In one embodiment, after determining the mean absolute error (MAE), labels can also be determined. Based on the difference between the MAE and the labels, the parameters of the adapter and value head of the review model are adjusted in a direction that reduces this difference. The labels are generated based on the first rating result and the predicted response content. For example, they can be based on a formula... Generate tags. Among them, The first score result These are preset parameters. This is the result of the divergence calculation. The divergence calculation result characterizes the difference between the predicted response and the response output by the base model.

[0121] In this embodiment, since the parameters of the comment model are adjusted based on the average absolute error between the first and second rating results, the comment model can continuously fit the rating of the reward model, thereby realizing the perception of the current model's capabilities and preventing the language model from getting stuck in a bottleneck.

[0122] In one embodiment, determining the language model loss based on the predicted response content, the first scoring result, and the second scoring result includes: determining the overall response score based on the first scoring result and the second scoring result; determining the penalty loss based on the difference between the predicted response content and the response content output by the base model based on the first model instruction; and determining the language model loss based on the overall response score and the penalty loss.

[0123] Specifically, when it is necessary to adjust the parameters of the language model, refer to Figure 10 The computer device can determine the overall response score based on the first and second scoring results. For example, the computer device can add the first and second scoring results to obtain the overall response score. Furthermore, to ensure that the distribution of the language model output does not differ too much from the output distribution of the base model, a penalty loss can be determined based on the difference between the predicted response content and the response content output by the base model based on the instructions of the first model. Once the overall response score and penalty loss are determined, the language model loss can be determined based on these two factors. For example, the overall response score can be added to the penalty loss to obtain the language model loss, and the parameters of the language model's adapter and value head can be adjusted based on the language model loss. Figure 10 A schematic diagram illustrating the determination of language model loss in one embodiment is shown.

[0124] In this embodiment, by determining the penalty term, the content output by the language model can be made to be similar to that output by the base model, thereby avoiding large deviations in the output results of the language model.

[0125] In one embodiment, determining the penalty term loss based on the difference between the predicted response content and the response content output by the base model based on the first model instruction includes: obtaining the first logarithmic probability corresponding to each word element in the predicted response content output by the language model to be trained in the current round; concatenating the first model instruction and the predicted response content to obtain a concatenation instruction; inputting the concatenation instruction into the base model and calling the base model to output the second logarithmic probability corresponding to each word element in the predicted response content; and determining the penalty term loss based on the first logarithmic probability and the second logarithmic probability.

[0126] Specifically, when determining the penalty loss, the computer device can obtain the first logarithmic probability set synchronously output by the language model to be trained in the current round when outputting the predicted response content. This first logarithmic probability set includes the logarithmic probability corresponding to each dimension in the predicted response content. In a probabilistic language model, the model calculates the probability of each token in the input text, representing the likelihood of that token appearing given the preceding context. The logarithmic probability of each token is the natural logarithm of this probability value. For ease of description, the logarithmic probability output by the language model to be trained in the current round will be referred to as the first logarithmic probability.

[0127] Furthermore, the computer device can concatenate the first model instruction with the predicted response content to obtain a concatenation instruction, and input the concatenation instruction into the base model so that the base model outputs the log probability corresponding to each word in the predicted response content. For ease of description, the log probability output by the base model will be referred to as the second log probability below. The computer device can determine the penalty loss based on the first log probability and the second log probability. For example, the computer device can subtract the second log probability from the first log probability corresponding to the same word to obtain the probability difference, and add the probability differences corresponding to each word to obtain the penalty term loss.

[0128] In this embodiment, the penalty loss is determined by logarithmic probability, which makes the determined penalty loss more accurate, and thus the language model trained based on the penalty loss is also more accurate.

[0129] In one embodiment, for a better understanding of this application, reference is made to... Figure 12 The following describes the overall training process of this application. Steps 1-2: Input the model prompts (i.e., the first model instructions) into the base model, and the base model outputs a reference response. Steps 3-4: Input the model prompts (i.e., the first model instructions) into the language model, and the language model outputs a predicted response. Step 5: Calculate the KL divergence between the reference response and the predicted response. Step 6: Input the predicted response into the reward model, and the reward model outputs a first rating result. The comment model then scores the predicted response to obtain a second rating result. Step 7: Generate labels based on the first rating result and the divergence calculation result, for example, using a formula... Generate tags. Among them, The first score result These are preset parameters. The result is the divergence calculation. Step 8. The label can be used as a reference, and the comment model is trained using the comment model loss calculated based on the first and second rating results. Step 9. The language model is trained using the language model loss calculated based on the first rating result, the second rating result, and the predicted response generation. Figure 12 A schematic diagram of the overall process of model training in one embodiment is shown.

[0130] Model training is a crucial step in improving model capabilities, but it also consumes the most training resources. Model training involves the interaction of four models: the language model, the comment model, the reward model, and the base model. Taking training Llama-7B as an example, all four models are 7B in size. Even without training, simply loading all four models into GPU memory requires at least 54GB (fp16), while commonly used GPUs only have 40GB. During training, the reward and base models have frozen parameters; only the reward and comment models are updated. This means the number of parameters trained reaches 14B (7B+7B). Therefore, before improvements, training a 7B model required at least 16-32 40GB GPUs to start training. However, this application, by sharing the base model with the language model, frequency model, and reward model, and using adapter parameters for each, allows training of a 7B model on a single 40GB GPU, significantly saving GPU resources.

[0131] In one embodiment, the method further includes: obtaining the first score result output by the reward model after training in multiple consecutive training rounds; stopping the training of the language model when the difference between the obtained first score results is less than or equal to a preset difference threshold, thereby obtaining the language model after multiple training rounds.

[0132] Specifically, when the difference between the first score results output by the reward model after training in multiple consecutive training rounds is less than or equal to a preset difference threshold, it can be determined that the first score result obtained by the predicted response content output by the language model has become stable. At this point, it can be considered that the training stopping condition has been met, and the training of the language model is stopped to obtain the language model after multiple training rounds.

[0133] In one embodiment, the language model trained through multiple rounds can be used to output text or dialogue content. The text generated by the multi-round trained language model is more grammatically correct, richer in detail, and better captures the core theme, resulting in a +2.2% improvement in human evaluation accuracy on the evaluation set. The answers output by the multi-round trained language model are more detailed and reasonable, and can correctly answer some tricky questions. On 11 subjective question-answering evaluation sets, the multi-round trained language model achieved a score of 10 wins, 1 draw, and 0 losses compared to the original model, and also achieved remarkable performance on objective natural language understanding tasks. As can be seen from the examples in Table 1 below, the answers output by the language model trained using the embodiments of this application are more detailed and reasonable, and can correctly answer some tricky questions.

[0134] Table 1: Result Comparison Table

[0135]

[0136] In one embodiment, the method further includes: displaying an icon corresponding to the language model after multiple rounds of training; displaying a content display interface in response to a trigger operation on the icon; the content display interface includes a model instruction input box and a response display area; in response to a model instruction input operation, inputting a third model instruction through the model instruction input box into the language model after multiple rounds of training, calling the language model after multiple rounds of training to output the response content corresponding to the third model instruction; and displaying the response content corresponding to the third model instruction in the response display area.

[0137] Specifically, after obtaining the language model through multiple rounds of training, the terminal can display an icon corresponding to the trained language model. When the user clicks this icon, the terminal can display the content interface. (Reference) Figure 11 The content display interface includes a model instruction input box 1101 and a response display area 1102. Users can input third model instructions in the model instruction input box, so that the language model trained in multiple rounds can output corresponding response content based on the third model instructions, and then the terminal can display the response content in the response display area. Figure 11 A schematic diagram of a content display interface in one embodiment is shown.

[0138] In this embodiment, by displaying a content display page, users can interact with the trained language model through the content display page, thereby improving interaction efficiency.

[0139] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0140] This application also provides an application scenario in which the above-described model training method is applied. Specifically, the model training method is applied in this scenario as follows:

[0141] Before using a language model to output text, you can train the language model first. When training the language model is required, refer to [reference needed]. Figure 13 The approach involves first training a base model, followed by a reward model. This results in a reward model capable of accurately scoring the output of any model, thus replacing human scoring (human feedback). Subsequently, a comment model and a language model are trained. By combining the language model's output with the reward model's scoring, reinforcement learning is used to continuously optimize model parameters, increasing the reward or score for the language model's responses. Reward model training utilizes positive and negative sample pairs, significantly impacting the final training outcome. The reward model acts as a scorer, taking "model instructions + response content" as input and outputting a score—a rating of the model's generated content. Higher scores indicate greater alignment with human preferences, such as realism, accuracy, and safety. This application appends a value head, a layer of MLP, to the base model, compressing high-dimensional vectors into a single output value. The reward model's training parameters are added via an adapter, requiring only a small number of parameter updates during training. This step is fast, cost-effective, and requires minimal GPU resources.

[0142] The goal of training the comment model and language model is to continuously improve the score of the language model's responses, thus realizing a link between human feedback and model generation optimization. The language model undergoes multi-round training: the iterative process is as follows: First, given the first model instruction, the language model generates corresponding predicted responses. The reward model scores the responses based on the first model instruction and the predicted responses. Then, using the first model instruction, the predicted responses, and the scores, the parameters of the adapters in the comment model and language model are trained: the comment model continuously fits the score from the reward model to perceive its current capabilities and prevent the language model from hitting a bottleneck; the language model uses policy gradients to update the adapter parameters, continuously optimizing towards higher scores. There is also a penalty term to ensure that the distribution of the language model's output does not differ too much from the distribution of the base model's output, avoiding significant deviations in the output results. Through continuous iterative optimization, the rewards obtained by the language model's generated results become increasingly higher, and the generated results increasingly conform to human preferences.

[0143] The specific training process for comment models and language models can be roughly divided into the following steps:

[0144] Model initialization: The language model is initialized to the fine-tuned base model, and the adapter parameters are set to 0. The comment model initializes the adapter parameters of the language model.

[0145] Result generation: Input the prepared first model instruction, and the language model generates the result, which usually generates a batch of instruction results.

[0146] Scoring: Use a reward model to score the batch of generated results.

[0147] Reward model iteration: For each lexical element of the predicted response, calculate the K-divergence (i.e., penalty term) of the probability distribution between the language model and the base model to ensure that the language model does not deviate excessively from the fine-tuned model. Then, calculate the overall response score based on the scores from the comment model and the reward model. Add the K-divergence and the overall response score together, and then update the language model parameters using policy gradient descent.

[0148] The review model iterates by calculating the average absolute error between its own score and the reward score as the training loss and updating the review model parameters.

[0149] The language model and comment model iterate multiple times to minimize the loss based on batch instruction data, and then enter the next round of the loop. This process is repeated for a certain number of steps until the reward model score generated by the language model tends to stabilize or the maximum number of steps is reached, at which point training stops.

[0150] During the training of the comment model and the language model, the basic model parameters of the language model, comment model, and reward model are shared. Together with the basic model itself, this constitutes the four models involved in the multi-round training process of the language model. In addition, the adapter parameters corresponding to the reward model are frozen during training. Only the adapter parameters corresponding to the reward model and the comment model need to be updated during training. In this way, the model training can be completed on a 40G version graphics card.

[0151] Once trained, the language model can be used to generate copy. Figure 13 A schematic diagram of the overall process of model training is shown in another embodiment.

[0152] The above application scenarios are merely illustrative. It is understood that the application of the model training methods provided in the embodiments of this application is not limited to the above scenarios.

[0153] Based on the same inventive concept, this application also provides a model training apparatus for implementing the model training method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more model training apparatus embodiments provided below can be found in the limitations of the model training method described above, and will not be repeated here.

[0154] In one embodiment, such as Figure 14 As shown, a model training device 1400 is provided, including: a model acquisition module 1402, a reward model training module 1404, and a joint training module 1406, wherein:

[0155] The model acquisition module 1402 is used to acquire the comment model, language model, and reward model; the comment model, language model, and reward model each include a base model and an adapter with fewer parameters than the base model.

[0156] The reward model training module 1404 is used to train the reward model based on preset positive and negative sample pairs to obtain the trained reward model.

[0157] The joint training module 1406 is used to train the comment model and the language model in multiple rounds based on the trained reward model. In the current round of the multi-round training process, a preset first model instruction is input into the language model to be trained in the current round to obtain the predicted response content corresponding to the first model instruction. The trained reward model and the comment model to be trained in the current round are called to score the predicted response content respectively to obtain a first score result and a second score result. Based on the first score result and the second score result, the parameters of the adapters of the comment model and the language model to be trained in the current round are adjusted respectively to obtain the comment model and the language model to be trained in the next round.

[0158] In one embodiment, the model acquisition module 1402 is further configured to acquire a base model, an adapter, and a value head; the value head refers to a machine learning module used to evaluate the value of the current state; a language model is generated based on the base model and the adapter; and a reward model and a comment model are generated based on the base model, the adapter, and the value head.

[0159] In one embodiment, the model acquisition module 1402 is further configured to acquire the weight matrix of the base model and determine the dimension of the weight matrix; generate a low-rank matrix based on the dimension of the weight matrix; wherein the dimension of the low-rank matrix is ​​less than the dimension of the weight matrix; and generate an adapter based on the low-rank matrix.

[0160] In one embodiment, the reward model training module 1404 is further configured to acquire positive and negative sample pairs, which include positive samples and negative samples; positive samples include second model instructions and high-quality response content generated based on the second model instructions that meets the high-quality response conditions; negative samples include second model instructions and low-quality response content generated based on the second model instructions that does not meet the high-quality response conditions; positive samples are input into the reward model, and the reward model is invoked to score the high-quality response content based on the second model instructions to obtain positive sample scoring results; negative samples are input into the reward model, and the reward model is invoked to score the low-quality response content based on the second model instructions to obtain negative sample scoring results; the parameters of the reward model are adjusted according to the positive sample scoring results and the negative sample scoring results to obtain the trained reward model.

[0161] In one embodiment, the reward model includes a base model, a value head, and an adapter; the reward model training module 1404 is further configured to determine the score difference between positive sample scoring results and negative sample scoring results; fix the parameters of the base model in the reward model, and adjust the parameters of the value head and adapter in the reward model in the direction of increasing the score difference to obtain the trained reward model.

[0162] In one embodiment, the joint training module 1406 is further configured to synchronize the parameters of the adapter of the trained reward model to the adapter of the comment model to obtain the comment model to be trained in the first round; obtain preset values ​​and set the parameters of the adapter in the language model to the preset values ​​to obtain the language model to be trained in the first round.

[0163] In one embodiment, the joint training module 1406 is further configured to determine the comment model loss based on the first rating result and the second rating result, and adjust the parameters of the adapter of the comment model to be trained in the current round based on the comment model loss; and determine the language model loss based on the predicted response content, the first rating result and the second rating result, and adjust the parameters of the adapter of the language model to be trained in the current round based on the language model loss.

[0164] In one embodiment, the first model instruction is located in the first model instruction set; in one embodiment, the joint training module 1406 is further configured to, when the language model to be trained in the current round outputs multiple predicted response contents based on multiple first model instructions in the model instruction set, call the trained reward model and the comment model to be trained in the current round to score each predicted response respectively, to obtain multiple first score results and multiple second score results; determine the mean absolute error between the multiple first score results and the multiple second score results; and determine the comment model loss based on the mean absolute error.

[0165] In one embodiment, the joint training module 1406 is further configured to determine the overall response score based on the first scoring result and the second scoring result; determine the penalty term loss based on the difference between the predicted response content and the response content output by the base model based on the first model instruction; and determine the language model loss based on the overall response score and the penalty term loss.

[0166] In one embodiment, the joint training module 1406 is further configured to obtain the first logarithmic probability corresponding to each word in the predicted response content output by the language model to be trained in the current round; concatenate the first model instruction and the predicted response content to obtain a concatenation instruction; input the concatenation instruction to the base model, call the base model to output the second logarithmic probability corresponding to each word in the predicted response content; and determine the penalty term loss based on the first logarithmic probability and the second logarithmic probability.

[0167] In one embodiment, the joint training module 14606 is further configured to obtain the first score result output by the reward model after training in multiple consecutive training rounds; when the difference between the obtained first score results is less than or equal to a preset difference threshold, the training of the language model is stopped to obtain the language model after multiple training rounds.

[0168] In one embodiment, the model training device 1400 further includes using a model to display icons corresponding to the language model after multiple rounds of training; in response to a trigger operation on the icons, displaying a content display interface; the content display interface includes a model instruction input box and a response display area; in response to a model instruction input operation, inputting a third model instruction through the model instruction input box into the language model after multiple rounds of training, calling the language model after multiple rounds of training to output the response content corresponding to the third model instruction; and displaying the response content corresponding to the third model instruction in the response display area.

[0169] Each module in the aforementioned model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0170] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 15 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores model training data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a model training method.

[0171] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 16As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a model training method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0172] Those skilled in the art will understand that Figure 15 and Figure 16 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0173] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0174] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0175] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0176] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0177] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, etc., and are not limited to these.

[0178] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A model training method, characterized in that, The method includes: Obtain a comment model, a language model, and a reward model; the comment model, the language model, and the reward model each include a base model and an adapter with fewer parameters than the base model; The reward model is trained based on a preset pair of positive and negative samples to obtain the trained reward model. The comment model and the language model are trained in multiple rounds based on the trained reward model. In the current round of the training process, a preset first model instruction is input into the language model to be trained in the current round to obtain the predicted response content corresponding to the first model instruction. The trained reward model and the comment model to be trained in the current round are used to score the predicted response content to obtain a first score result and a second score result. Based on the first and second scoring results, the parameters of the adapters for the comment model and language model to be trained in the current round are adjusted to obtain the comment model and language model to be trained in the next round.

2. The method according to claim 1, characterized in that, The obtained comment model, language model, and reward model include: Obtain the base model, adapter, and value head; the value head refers to the machine learning module used to evaluate the value of the current state. Based on the base model and the adapter, a language model is generated; Based on the base model, the adapter, and the value head, a reward model and a comment model are generated.

3. The method according to claim 2, characterized in that, The method further includes: Obtain the weight matrix of the base model and determine the dimension of the weight matrix; A low-rank matrix is ​​generated based on the dimension of the weight matrix; wherein the dimension of the low-rank matrix is ​​less than the dimension of the weight matrix. The adapter is generated based on the low-rank matrix.

4. The method according to claim 1, characterized in that, The step of training the reward model based on preset positive and negative sample pairs to obtain the trained reward model includes: Obtain positive and negative sample pairs, wherein the positive and negative sample pairs include positive samples and negative samples; the positive samples include the second model instruction and high-quality response content generated based on the second model instruction that meets the high-quality response conditions; the negative samples include the second model instruction and low-quality response content generated based on the second model instruction that does not meet the high-quality response conditions; The positive samples are input into the reward model, and the reward model is invoked to score the high-quality response content based on the second model instruction to obtain the positive sample scoring results; The negative sample is input into the reward model, and the reward model is invoked to score the low-quality response content based on the second model instruction to obtain the negative sample score result; Based on the positive sample scoring results and the negative sample scoring results, the parameters of the reward model are adjusted to obtain the trained reward model.

5. The method according to claim 4, characterized in that, The reward model includes a base model, a value head, and an adapter; adjusting the parameters of the reward model based on the positive sample scoring results and the negative sample scoring results to obtain the trained reward model includes: Determine the score difference between the positive sample scoring results and the negative sample scoring results; By fixing the parameters of the base model in the reward model, the parameters of the value head and adapter in the reward model are adjusted in the direction of increasing the score difference to obtain the trained reward model.

6. The method according to claim 1, characterized in that, Prior to the first training round in the multi-round training process, the method further includes: The parameters of the adapter of the trained reward model are synchronized to the adapter of the comment model to obtain the comment model to be trained in the first round. Obtain preset values ​​and set the parameters of the adapter in the language model to the preset values ​​to obtain the language model to be trained in the first round.

7. The method according to claim 1, characterized in that, The step of adjusting the parameters of the adapters for the comment model and the language model to be trained in the current round, based on the first and second scoring results, includes: Based on the first rating result and the second rating result, the comment model loss is determined, and the parameters of the adapter of the comment model to be trained in the current round are adjusted based on the comment model loss; Based on the predicted response content, the first rating result, and the second rating result, the language model loss is determined, and the parameters of the adapter for the language model to be trained in the current round are adjusted based on the language model loss.

8. The method according to claim 7, characterized in that, The first model instruction is located in the first model instruction set; the step of calling the trained reward model and the comment model to be trained in the current round to score the predicted response content, and obtaining a first score result and a second score result, includes: When the language model to be trained in the current round outputs multiple predicted response contents based on multiple first model instructions in the model instruction set, the trained reward model and the comment model to be trained in the current round are called to score each predicted response, thereby obtaining multiple first score results and multiple second score results; The step of determining the review model loss based on the first rating result and the second rating result includes: Determine the average absolute error between the plurality of first scoring results and the plurality of second scoring results; The comment model loss is determined based on the mean absolute error.

9. The method according to claim 7, characterized in that, The step of determining the language model loss based on the predicted response content, the first rating result, and the second rating result includes: The overall response score is determined based on the first and second scoring results. The penalty loss is determined based on the difference between the predicted response content and the response content output by the base model based on the instructions of the first model; The language model loss is determined based on the overall response score and the penalty term loss.

10. The method according to claim 9, characterized in that, The step of determining the penalty loss based on the difference between the predicted response content and the response content output by the base model based on the instructions of the first model includes: Obtain the first logarithmic probability of each word element in the predicted response output of the language model to be trained in the current round; The first model instruction and the predicted response content are concatenated to obtain the concatenation instruction; The concatenation instruction is input into the basic model, and the basic model is called to output the second log probability corresponding to each word element in the predicted response content; The penalty term loss is determined based on the first logarithmic probability and the second logarithmic probability.

11. The method according to claim 1, characterized in that, The method further includes: Obtain the first score result output by the trained reward model in multiple consecutive training rounds; When the difference between the obtained first score results is less than or equal to the preset difference threshold, the training of the language model is stopped, and the language model after multiple rounds of training is obtained.

12. The method according to any one of claims 1 to 10, characterized in that, The method further includes: Displays icons corresponding to the language model after multiple rounds of training; In response to a trigger operation on the icon, a content display interface is displayed; the content display interface includes a model instruction input box and a response display area; In response to the model instruction input operation, the third model instruction entered through the model instruction input box is input into the language model after multiple rounds of training, and the language model after multiple rounds of training is called to output the response content corresponding to the third model instruction; The response content corresponding to the third model instruction is displayed in the response display area.

13. A model training device, characterized in that, The device includes: The model acquisition module is used to acquire a comment model, a language model, and a reward model; the comment model, the language model, and the reward model each include a base model and an adapter with fewer parameters than the base model. The reward model training module is used to train the reward model based on preset positive and negative sample pairs to obtain the trained reward model. The joint training module is used to train the comment model and the language model in multiple rounds based on the trained reward model. In the current round of the multi-round training process, a preset first model instruction is input into the language model to be trained in the current round to obtain the predicted response content corresponding to the first model instruction. The trained reward model and the comment model to be trained in the current round are used to score the predicted response content to obtain a first score result and a second score result. Based on the first score result and the second score result, the parameters of the adapters of the comment model and the language model to be trained in the current round are adjusted to obtain the comment model and the language model to be trained in the next round.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images