Model training method, device and equipment and readable storage medium

Abstract

The invention provides a model training method, device and equipment and a readable storage medium, quality scoring is performed on positive and negative samples in sample data, quality score differences of the positive and negative samples are fused into a loss function, and the loss function is designed by combining score differences of the positive and negative samples, so that the quality of the positive and negative samples is improved. Therefore, the model can perceive the quality difference of the positive and negative samples more clearly in the training process, and the explicit score difference is utilized to guide the model to be more likely to amplify the difference between the high-quality positive sample and the low-quality negative sample during parameter updating, so that the model generation quality and the training effect are improved.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a model training method, apparatus, device, and readable storage medium. Background Art

[0002] Currently, when training models using the DPO (Direct Preference Optimization) training strategy, the constructed loss function model usually optimizes the output results only through a simple comparison of positive and negative examples. However, this approach makes it difficult for the model to fully capture the significant differences between positive and negative examples during training, resulting in an inability to effectively widen the gap between positive and negative examples, affecting the model's training effectiveness. Summary of the Invention

[0003] In view of this, in order to solve the above technical problems, the present application provides a model training method, device, equipment and readable storage medium.

[0004] Specifically, this application is implemented through the following technical solutions:

[0005] According to a first aspect of an embodiment of the present application, a model training method is provided, the method comprising:

[0006] When training the model using the direct preference optimization (DPO) training method, for each input prompt, positive example, and negative example included in each sample data used in this round of training, obtain the first quality score of the positive example and the second quality score of the negative example in the sample data;

[0007] Obtaining a first probability predicted by the model for the positive example and a second probability predicted for the negative example; the first probability represents the probability that a result of the model processing the input prompt is the positive example; the second probability represents the probability that a result of the model processing the input prompt is the negative example;

[0008] Based on the first quality score and the second quality score, weighting the first probability and the second probability to determine the preference difference loss corresponding to the sample data;

[0009] According to the preference difference loss corresponding to each sample data, as well as the supervised fine-tuning SFT loss of the positive samples and the SFT loss of the negative samples in the sample data, the loss of all sample data used in this round of training is constructed as the target loss of the model in this round of training, so as to adjust the parameters of the model based on the target loss.

[0010] Optionally, constructing the loss of all sample data used in this round of training includes:

[0011] Using the three preset hyperparameters as weights, the preference difference loss, the SFT loss of the positive sample, and the SFT loss of the negative sample corresponding to the sample data are weighted and summed to obtain the weighted loss of the sample data;

[0012] The weighted loss of each sample data used in this round of training is averaged to obtain the loss of all sample data used in this round of training.

[0013] Optionally, the weighting the first probability and the second probability based on the first quality score and the second quality score includes:

[0014] Based on the probability distribution normalization results of the first probability and the second probability, a first preference probability and a second preference probability are determined; the first preference probability is used to reflect the preference strength of the model for the positive example; the second preference probability is used to reflect the preference strength of the model for the negative example;

[0015] The first preference probability and the second preference probability are weighted based on the first quality score and the second quality score.

[0016] Optionally, the probability distribution normalization processing is implemented by normalization; and the determining of the first preference probability and the second preference probability based on the probability distribution normalization processing results of the first probability and the second probability includes:

[0017] Normalizing the first probability and the second probability corresponding to the sample data by a softmax function to obtain normalized first probability and second probability;

[0018] The logarithm of the normalized first probability is used as the first preference probability, and the logarithm of the normalized second probability is used as the second preference probability.

[0019] Optionally, the weighting the first preference probability and the second preference probability includes:

[0020] Multiplying a first quality score corresponding to a positive example in the sample data by a first preference probability corresponding to the positive example to obtain a first result;

[0021] Multiplying the second quality score corresponding to the negative sample in the sample data by the second preference probability corresponding to the negative sample to obtain a second result;

[0022] The negative of the sum of the first result and the second result is used as the preference difference loss corresponding to the sample data.

[0023] Optionally, obtaining a first quality score of a positive example and a second quality score of a negative example in the sample data includes:

[0024] Using the trained preference ranking model PRM, quality scores are respectively performed on the positive samples and negative samples included in the sample data to obtain a first quality score corresponding to the positive sample and a second quality score corresponding to the negative sample;

[0025] The quality score range of the PRM score is [-a, a], where a is a positive integer; the first quality score in the same sample data is greater than the second quality score.

[0026] Optionally, the method further includes:

[0027] The negative logarithm result of the first probability corresponding to the positive example is used as the SFT loss of the positive example, and the negative logarithm result of the second probability corresponding to the negative example is used as the SFT loss of the negative example.

[0028] According to a second aspect of an embodiment of the present application, a model training device is provided, the device comprising:

[0029] A quality score acquisition module is used to acquire, when training the model using the DPO training method, a first quality score of the positive sample in the sample data and a second quality score of the negative sample in the sample data for each input prompt, positive sample, and negative sample included in the current round of training;

[0030] a model prediction acquisition module, configured to acquire a first probability predicted by the model for the positive example and a second probability predicted for the negative example; the first probability representing the probability that the result of the model processing the input prompt is the positive example; and the second probability representing the probability that the result of the model processing the input prompt is the negative example;

[0031] a preference difference loss determination module, configured to perform weighted processing on the first probability and the second probability based on the first quality score and the second quality score, to determine a preference difference loss corresponding to the sample data;

[0032] The target loss construction module is used to construct the loss of all sample data used in this round of training as the target loss of the model in this round of training based on the preference difference loss corresponding to each sample data, the supervised fine-tuning SFT loss of the positive samples in the sample data, and the SFT loss of the negative samples, so as to adjust the parameters of the model based on the target loss.

[0033] Optionally, the target loss construction module is specifically used to:

[0034] Using the three preset hyperparameters as weights, the preference difference loss, the SFT loss of the positive sample, and the SFT loss of the negative sample corresponding to the sample data are weighted and summed to obtain the weighted loss of the sample data;

[0035] The weighted loss of each sample data used in this round of training is averaged to obtain the loss of all sample data used in this round of training.

[0036] Optionally, the preference difference loss determination module includes:

[0037] a preference probability determination module, configured to determine a first preference probability and a second preference probability based on a probability distribution normalization processing result of the first probability and the second probability; the first preference probability is used to reflect the strength of the model's preference for the positive example; the second preference probability is used to reflect the strength of the model's preference for the negative example;

[0038] A weighted processing module is configured to perform weighted processing on the first preference probability and the second preference probability based on the first quality score and the second quality score.

[0039] Optionally, the probability distribution standardization process is implemented by normalization; and the preference probability determination module is specifically configured to:

[0040] Normalizing the first probability and the second probability corresponding to the sample data by a softmax function to obtain normalized first probability and second probability;

[0041] The logarithm of the normalized first probability is used as the first preference probability, and the logarithm of the normalized second probability is used as the second preference probability.

[0042] Optionally, the weighted processing module is specifically configured to:

[0043] Multiplying a first quality score corresponding to a positive example in the sample data by a first preference probability corresponding to the positive example to obtain a first result;

[0044] Multiplying the second quality score corresponding to the negative sample in the sample data by the second preference probability corresponding to the negative sample to obtain a second result;

[0045] The negative of the sum of the first result and the second result is used as the preference difference loss corresponding to the sample data.

[0046] Optionally, the quality score acquisition module is specifically configured to:

[0047] Using the trained preference ranking model PRM, the positive samples and negative samples included in the sample data are respectively scored to obtain a first quality score corresponding to the positive sample and a second quality score corresponding to the negative sample; wherein the quality score range of the PRM score is [-a, a], where a is a positive integer; the first quality score in the same sample data is greater than the second quality score.

[0048] Optionally, the device further comprises:

[0049] The negative logarithm result of the first probability corresponding to the positive example is used as the SFT loss of the positive example, and the negative logarithm result of the second probability corresponding to the negative example is used as the SFT loss of the negative example.

[0050] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory and a processor; the memory is used to store a computer program; and the processor is used to execute the above-mentioned model training method by calling the computer program.

[0051] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned model training method is implemented.

[0052] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0053] In the technical solution provided in the above-mentioned application, quality scores are given to the positive and negative examples in the sample data, and the difference in the quality scores of the positive and negative examples is incorporated into the loss function. By designing the loss function in combination with the score difference of the positive and negative examples, the model can more clearly perceive the quality difference between the positive and negative samples during the training process, and use the explicit score difference to guide the model to be more inclined to amplify the gap between high-quality positive examples and low-quality negative examples when updating parameters, thereby improving the model generation quality and training effect.

[0054] It should be understood that the above general description and the detailed description below are merely exemplary and explanatory and cannot limit the present application. In addition, any embodiment in the present application does not necessarily achieve all the effects described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0056] Figure 1 This is a flow chart of a model training method shown in an exemplary embodiment of the present application;

[0057] Figure 2This is a process of constructing a target loss function for this round of training of a model as shown in an exemplary embodiment of the present application;

[0058] Figure 3 This is a schematic diagram of a step of determining a first preference probability and a second preference probability, shown in an exemplary embodiment of the present application;

[0059] Figure 4 1 is a schematic diagram of a PRM model scoring positive and negative samples, shown in an exemplary embodiment of the present application;

[0060] Figure 5 It is a structural diagram of a model training device shown in an exemplary embodiment of the present application;

[0061] Figure 6 It is a hardware schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0062] Here, exemplary embodiments will be described in detail, with examples shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Instead, they are merely examples of devices and methods consistent with certain aspects of this application as detailed in the appended claims. It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other.

[0063] DPO (Direct Preference Optimization) is a method for training models that aims to directly optimize the model's output preferences through human feedback without explicitly training the reward model. The core idea is to use contrastive learning to allow the model to learn to distinguish between "good" and "bad" outputs, thereby directly optimizing the model's preferences. DPO training uses a preference dataset, in which each sample data contains different outputs for the same input prompt, and labels which outputs are "good" as positive examples (preferred) and which are "bad" outputs as negative examples (not preferred). The input prompt is the input text of the model, which can be a user's question or task description; the positive example represents the output that humans think is better for the input prompt; the negative example represents the output that humans think is worse for the same input prompt.

[0064] For example, a sample data includes:

[0065] {Input prompt: "Please explain what quantum computing is.",

[0066] Positive example: "Quantum computing is a computing method based on the principles of quantum mechanics. It uses the superposition and entanglement of quantum bits (qubits) to process multiple computing tasks simultaneously, thereby achieving exponential speedups on certain problems."

[0067] Negative example: "Quantum computing is a computer that is faster than a normal computer and uses quantum computers."

[0068] }

[0069] DPO training uses a contrastive learning approach to train the model so that the probability of preferred outputs is higher and the probability of unpreferred outputs is lower. By maximizing the probability ratio between preferred and unpreferred outputs, the model's preference is directly optimized. Currently, when training models using the DPO training strategy, the model typically optimizes the output results by simply comparing positive and negative examples. For example, the following formula is an exemplary DPO loss function, which achieves preference optimization by maximizing the ratio of the probability of positive examples to the probability of negative examples:

[0070]

[0071] Where x is the input prompt in the sample data, yw is the positive example (high-quality answer) in the sample data, and yl is the negative example (low-quality answer) in the sample data; π θ is the model that needs to be trained using the DPO training strategy; ref is the reference model (usually the initial model after supervised fine-tuning), which is usually the same as the current model π θ The initial parameters of are consistent, providing a probability benchmark and regularization constraints; β is the temperature coefficient, which is used to control the optimization intensity; σ is the sigmoid function.

[0072] This loss function directly optimizes the policy model πθ by comparing the generation probability of positive and negative samples, making it more inclined to generate high-quality answers yw while suppressing low-quality answers yl. The loss function encourages πθ to have an advantage over positive samples. As much as possible to have an advantage over negative examples By comparing the logarithmic advantage difference between positive and negative samples and combining it with the constraints of the reference model, the model's generation probability of positive samples is significantly higher than that of the reference model, while the generation probability of negative samples is significantly lower than that of the reference model, thus achieving model parameter adjustment.

[0073] However, under the above-mentioned method of achieving preference optimization by maximizing the ratio of the probability of positive samples to the probability of negative samples, the model cannot fully learn the significant differences between positive and negative samples during training, resulting in the inability to effectively widen the gap between positive and negative samples, affecting the training effect of the model.

[0074] In view of this, the present application provides a model training method suitable for the DPO training process of the model. The method scores the quality of positive and negative samples in the sample data, and incorporates the difference in the quality scores of positive and negative samples into the loss function. By designing the loss function based on the score difference of positive and negative samples, the model can more clearly perceive the quality difference between positive and negative samples during the training process, and use the explicit score difference to guide the model to be more inclined to amplify the gap between high-quality positive samples and low-quality negative samples when updating parameters, thereby improving the model generation quality and training effect.

[0075] See also Figure 1 As shown, the model training method provided in this application may include at least the following steps:

[0076] S101, when training a model using a direct preference optimization (DPO) training method, for each input prompt, positive example, and negative example included in each sample data used in this round of training, obtain a first quality score of the positive example and a second quality score of the negative example in the sample data;

[0077] The model can be a pre-trained or task-specific generative model, such as a language model, that can generate text, images, or other forms of data based on input prompts. During DPO training, the model needs to learn how to generate outputs that are more in line with human preferences based on input prompts.

[0078] In DPO training, each sample data includes an input prompt (prompt, the input text that the model needs to process, such as the user's question or task description), a positive example (positive example, the output that humans consider to be of higher quality or more in line with expectations for the same input prompt), and a negative example (negative example, the output that humans consider to be of lower quality or not in line with expectations for the same input prompt). The positive example and negative example are a pair of outputs of different qualities obtained by processing the input prompt.

[0079] The quality score represents an indicator used to quantify the quality of the answers to positive and negative samples. Specifically, the first quality score (i.e., the quality score of the positive sample) represents the quality of the answer to the positive sample. Similarly, the second quality score (i.e., the quality score of the negative sample) represents the quality of the answer to the negative sample. Answer quality assessment methods such as manual labeling and automatic model evaluation can be used to perform quality scoring on positive and negative samples. The quality scores of positive and negative samples can clearly reflect the quality difference between positive and negative samples, and provide a basis for the model to learn to distinguish between high-quality and low-quality outputs during the DPO training process. Based on the fact that positive samples are outputs that are more in line with human preferences and expectations, for the same input prompt, the first quality score of the positive sample should be greater than the second quality score of the negative sample.

[0080] Regarding the acquisition of the first / second quality scores, manual labeling, that is, having human annotators score positive and negative samples, can be used. For example, annotators can score positive and negative samples separately based on dimensions such as output accuracy, fluency, and relevance. The resulting quality scores can directly reflect human preferences and are highly accurate. Alternatively, model automatic evaluation can be used, such as using pre-trained BLEU, ROUGE, GPT-4, PRM, etc. to score positive and negative samples, achieving low-cost and high-efficiency sample quality scoring. In addition, sample quality scoring can also be achieved by combining manual labeling and automatic evaluation. For example, first use automatic evaluation to perform preliminary scoring on positive and negative samples in the sample data, and then use manual labeling to perform detailed scoring on a small amount of key data.

[0081] For example, some sample data is as follows:

[0082] ①Input prompt: Please write a short essay about environmental protection.

[0083] ② Positive example: Environmental protection is everyone's responsibility. We should reduce the use of disposable plastics, promote renewable energy, and protect wildlife habitats. Only in this way can we leave a beautiful planet for future generations.

[0084] ③ Negative example: Environmental protection is important, but I don’t know how to do it. Maybe I can drive less and plant more trees.

[0085] The positive example is scored using the pre-trained model, and the resulting first quality score is 0.8. The negative example is scored using the same method, and the resulting second quality score is -0.6.

[0086] S102, obtaining a first probability predicted by the model for the positive example and a second probability predicted for the negative example; the first probability represents the probability that the result of the model processing the input prompt is the positive example; the second probability represents the probability that the result of the model processing the input prompt is the negative example;

[0087] When training a model using the DPO method, the model needs to learn how to generate outputs that are more in line with human preferences based on given input prompts. During this process, the model evaluates the degree of match or similarity between the model output generated by processing the input prompts and different reference examples (such as positive examples and negative examples).

[0088] For each sample data point used in this round of training, the input prompts in the sample data are input into the model as the initial information or instructions received, which guide the model in generating outputs. Positive examples in the sample data are examples of outputs that the model is expected to generate during training, and the model needs to learn how to generate outputs similar to positive examples. Negative examples in the sample data are examples of outputs that the model is expected to avoid generating during training, and serve as negative examples for model optimization, so the model needs to learn how to avoid generating outputs similar to negative examples.

[0089] The model receives input prompts from sample data and processes them according to its internal parameters and algorithms, generating a probability distribution that represents the model's confidence in different possible outputs. The first probability represents the probability that the model's processing of the input prompts from the sample data will result in a positive example. That is, given the model's current network parameters, the model evaluates the similarity or degree of match between the processed input prompts and the positive example, reflecting the likelihood that the model will generate an output similar to the positive example. Similarly, the second probability represents the probability that the model's processing of the input prompts from the same sample data will result in a negative example. This represents the similarity or degree of match between the output generated by the model and the negative example, reflecting the likelihood that the model will generate an output similar to the negative example.

[0090] For example, for the i-th sample data, the input prompt x i and the positive example y choen Combine the inputs into the model and get the first probability f of the model output θ (x i ,y choen ), and similarly include the input prompt x i and negative examples y reject Combine the inputs into the model to get the second probability f of the model output θ (x i ,y reject ).

[0091] S103: Based on the first quality score and the second quality score, weight the first probability and the second probability to determine the preference difference loss corresponding to the sample data;

[0092] In this embodiment, the quality scores of positive and negative samples are combined with the reward tensor output by the model to form a dynamically weighted difference loss function based on the model's preference for positive and negative samples. This mechanism can automatically adjust the loss weight according to the difference in the quality scores of positive and negative samples: for positive and negative sample pairs with large score differences, the model will assign higher weights, thereby more significantly widening their representation differences; while for sample pairs with small score differences, the model will moderately adjust the weights to avoid overfitting. The dynamic weighting method can more precisely guide the direction of model optimization.

[0093] The obtained preference difference loss is used to reflect the difference in the degree to which the model tends to generate positive samples labeled as high quality and the degree to which it tends to generate negative samples labeled as low quality for a given prompt input, and quantify the model's ability to distinguish between positive / negative sample pairs. Therefore, for the first probability corresponding to the positive sample and the second probability corresponding to the negative sample in the sample data, the probability distribution can be first normalized based on the first probability and the second probability to quantify the model's preference for positive samples and negative samples, and then the first quality score and the second quality score are introduced into the process of constructing a loss function based on the difference in preference degree, and the first quality score is used as a weight for the model's preference degree for positive samples, and the second quality score is used as a weight for the model's preference degree for negative samples, so as to construct a preference difference loss that can reflect the difference in the model's preference strength for positive and negative samples in the sample data.

[0094] In some embodiments, when weighting the first probability and the second probability to determine the preference difference loss corresponding to the sample data, the first preference probability and the second preference probability can be first determined based on the probability distribution normalization processing results of the first probability and the second probability; wherein the obtained first preference probability is used to reflect the model's preference strength for the positive sample, and the second preference probability is used to reflect the model's preference strength for the negative sample. In comparison, the first probability / second probability reflects the model's preference for a single option, and the obtained first preference probability / second preference probability reflects the model's relative preference for positive and negative samples in the same sample data; further, based on the first quality score and the second quality score, the first preference probability and the second preference probability are weighted to construct the preference difference loss corresponding to the sample data.

[0095] Among them, the probability distribution normalization process is used to adjust the probability distribution of the model's output for positive and negative examples in the same sample data, so that it satisfies the requirement that the sum of the probabilities of a pair of positive and negative examples in the same sample data is 1 and the probability values ​​of the positive and negative examples are greater than or equal to 0. Because a direct relative comparison relationship exists between positive and negative examples in the same sample data after the probability distribution normalization process, the model's preference probability for different examples can be determined based on the probability distribution normalization results of the first and second probabilities, reflecting the degree to which the model tends to generate positive examples labeled as high-quality and negative examples labeled as low-quality for a given input prompt.

[0096] S104, based on the preference difference loss corresponding to each sample data, and the supervised fine-tuning SFT loss of the positive samples and the SFT loss of the negative samples in the sample data, construct the loss of all sample data used in this round of training as the target loss of the model in this round of training, so as to adjust the parameters of the model based on the target loss.

[0097] The supervised fine-tuning loss is used to measure the difference between the probability distribution predicted by the model and the true label. By minimizing this type of loss, the model can learn how to generate outputs that are more in line with human preferences. In this embodiment, the SFT loss of positive samples is used to ensure the model's fitting accuracy for high-quality samples, and the SFT loss of negative samples is used to control the model's ability to reject low-quality samples. The SFT losses of positive and negative samples can be designed based on the cross-entropy loss function. The SFT loss of positive samples is used to measure the difference between the first probability output by the model for positive samples and the ideal situation (i.e., the probability is 1). The SFT loss of negative samples is used to measure the difference between the first probability output by the model for positive samples and the ideal situation (i.e., the probability is 0). Based on the SFT losses of positive and negative samples, the model is encouraged to increase the predicted probability of positive samples to generate outputs that are more in line with human preferences, and the model is encouraged to reduce the predicted probability of negative samples to avoid generating outputs that do not conform to human preferences.

[0098] For example, the negative logarithm of the first probability corresponding to the positive sample can be used as the SFT loss of the positive sample, and the negative logarithm of the second probability corresponding to the negative sample can be used as the SFT loss of the negative sample. θ (x i ,y choen ), output the second probability f for the negative sample θ (x i ,y reject ), the corresponding SFT loss of the positive sample can be expressed as Lc = -log(f θ (x i ,y choen )), the SFT loss of negative samples can be expressed as Lr = -log(f θ (x i ,y reject )).

[0099] In this embodiment, the target loss of this round of iterative training of the model is composed of multiple loss terms, each loss term reflects the performance of the model in a specific aspect, and each loss term can be weighted based on the preference difference loss, the SFT loss of positive samples, and the SFT loss of negative samples to achieve a weighted combination of the preference difference loss based on the quality score and the SFT loss of positive and negative samples.

[0100] Based on the model's preference difference loss for a single sample data, the SFT loss for positive samples, and the SFT loss for negative samples, we can determine the comprehensive loss of a single sample data considering multiple loss items. For the comprehensive loss of all sample data used in this round of training, we can construct the target loss of the model for this round of training. This target loss reflects the overall performance of the model on the entire training set in this round of training.

[0101] After obtaining the target loss for this round of training, the gradient of the target loss with respect to the model parameters can be calculated using the backpropagation algorithm. The calculated gradient is then used to update the model parameters using gradient descent or its variants, such as stochastic gradient descent and Adam. This means that the parameters are updated in the opposite direction of the gradient (the direction in which the loss function decreases fastest) to minimize the target loss, thereby updating the model parameters for this round of training. The target loss combines multiple loss terms, including preference difference loss and SFT loss for positive and negative examples. Therefore, the parameter update process simultaneously considers the optimization objectives of these loss terms, enabling the model to generate outputs that are more in line with human preferences while maintaining high generation accuracy.

[0102] In an embodiment of the present disclosure, during the DPO training process of the model, quality scores are obtained for the positive and negative samples in the sample data respectively, and the quality scores are introduced as dynamic weights into the loss function construction that represents the preference difference of the model for positive and negative samples, so that the model automatically adjusts the degree of attention to samples with different reliability, and uses the preference difference loss to widen the quality score difference of positive and negative samples. The SFT loss of positive and negative samples is used to ensure the basic representation quality of positive and negative samples under supervised learning, and a target loss function that meets various training requirements is constructed. The model parameters are adjusted based on the target loss function, so that the model can more clearly perceive the quality difference of positive and negative samples during the training process, learn the distribution characteristics of positive and negative samples more comprehensively, alleviate the overfitting problem, improve the generalization performance, enhance the generation ability of the model while maintaining the stability of the generation task, thereby improving the model generation quality and training effect.

[0103] In some embodiments, for the aforementioned embodiment, the loss of all sample data used in this round of training can be constructed by weighting each loss item based on the preference difference loss, the SFT loss of positive samples, and the SFT loss of negative samples, thereby achieving a weighted combination of the preference difference loss based on quality scores and the SFT loss of positive and negative samples. Figure 2 The exemplary process of constructing the target loss function for this round of training of a model can be achieved through the following steps:

[0104] S201, using three preset hyperparameters as weights, performing a weighted summation of the preference difference loss corresponding to the sample data, the SFT loss of the positive sample, and the SFT loss of the negative sample to obtain a weighted loss for the sample data;

[0105] The three hyperparameters correspond to the weights of the preference difference loss, the SFT loss for positive samples, and the SFT loss for negative samples, respectively, and are used to balance the importance of different loss terms in the weighted loss of the sample data. The first hyperparameter, such as α, controls the importance of the preference difference loss in the target loss. Increasing α makes the model focus more on the consistency between the generated output and human preferences. The second hyperparameter, such as β, controls the importance of the supervised fine-tuning loss for positive samples in the target loss. Increasing β makes the model focus more on the accuracy of generating positive samples. The third hyperparameter, such as λ, controls the importance of the supervised fine-tuning loss for negative samples in the target loss. Increasing λ makes the model focus more on avoiding generating negative samples.

[0106] Hyperparameters can be selected based on experience or experimentation, or the optimal parameter combination can be found through methods such as grid search and random search. In some cases, hyperparameters can be adjusted dynamically during the training process to adapt to the learning needs of the model at different stages.

[0107] For example, the preference difference loss of a single sample data is expressed as L DPO , the SFT loss of the positive sample is Lc, and the SFT loss of the negative sample is Lr, then the weighted loss of the sample data can be expressed as L = α * L DPO +β*Lc+λ*Lr.

[0108] S202: Average the weighted loss of each sample data used in this round of training to obtain the loss of all sample data used in this round of training.

[0109] That is, the weighted losses of all sample data in this round of training are added and divided by the total number of sample data to get the average loss, which can be expressed as Among them, Li is the weighted loss of the i-th sample data, and N is the total number of sample data used in this round of training.

[0110] In the disclosed embodiment, a weighted loss of multiple loss items is considered for a single sample data through hyperparameters, and the weighted loss of each sample data used in this round of training is averaged to construct the target loss of this round of training. The obtained target loss can reflect the overall performance of the model on all sample data in this round of training. A single numerical value is obtained through averaging to quantify the performance of the model on the entire training set, smoothing out the extreme values ​​of individual sample data, so that the model pays more attention to the overall trend, reduces the impact of individual sample data anomalies or noise on model training, and improves the stability and robustness of training.

[0111] In some embodiments, the probability distribution normalization processing result based on the first probability of the positive sample and the second probability of the negative sample described in the above embodiment is used to determine the first preference probability and the second preference probability. The probability distribution normalization processing can be achieved by normalizing the result through the softmax function. That is, Figure 3 A schematic diagram illustrating an exemplary step of determining a first preference probability and a second preference probability:

[0112] S301, normalizing the first probability and the second probability corresponding to the sample data using a softmax function to obtain normalized first probability and second probability;

[0113] The Softmax function is a commonly used activation function used to convert a set of real numbers into a probability distribution. By applying the softmax function to the first probability and the second probability, it can be ensured that their sum is 1 and their relative size relationship remains unchanged. It is known that the normalization process of the softmax function can be expressed as:

[0114]

[0115] In this embodiment, y1 represents the first probability f of the positive sample. θ (x i ,y choen ), y2 represents the second probability f of the negative sample θ (x i ,y reject ), then the normalized first probability can be expressed as The normalized second probability can be expressed as

[0116] The normalized probability can more intuitively express the model's preference for positive and negative samples. A higher normalized probability indicates that the model is more inclined to select the sample. Therefore, the normalized first probability can be directly used as the model's first preference probability for positive samples, and the normalized second probability can be used as the model's second preference probability for negative samples.

[0117] Alternatively, calculations based on direct use of probability may lead to numerical instability. In order to avoid the problem of numerical underflow, combined with the fact that loss functions often involve logarithmic probabilities, step S302 can be further performed on the basis of step S301, and the logarithm result of the normalized first probability is used as the first preference probability, and the logarithm result of the normalized second probability is used as the second preference probability.

[0118] That is, the first preference probability can also be expressed as The second preference probability can be expressed as

[0119] In the disclosed embodiment, the preference probability after logarithmic transformation can more accurately quantify the model's preference strength for positive and negative samples. This quantification method can more delicately reflect the preference differences of the model between different samples. The preference difference loss function constructed based on the logarithmic preference probability can reduce the risk of overfitting due to anomalies or noise in individual sample data, and more effectively guide the model to learn the subtle differences in human preferences. By optimizing this loss function, the model can better fit the distribution of human preferences and improve the quality and conformity of the generated output.

[0120] In some embodiments, in the process of weighting the first preference probability and the second preference probability based on the first quality score and the second quality score, as shown in FIG. Figure 3 As shown, the first quality score corresponding to the positive example in the sample data can be used as the weight of the first preference probability, and the first quality score can be multiplied by the first preference probability corresponding to the positive example to obtain a first result. The second quality score corresponding to the negative example in the sample data can be used as the weight of the second preference probability, and the second quality score can be multiplied by the second preference probability corresponding to the negative example to obtain a second result. The negative of the sum of the first result and the second result can be used as the preference difference loss corresponding to the sample data.

[0121] For example, for the i-th sample data, the first quality score of the positive sample is Sci, and the second quality score of the negative sample is Sri, then the preference difference loss corresponding to the sample data can be expressed as:

[0122]

[0123] In the disclosed embodiment, by using the first quality score of the positive sample as the weight of the first preference probability and the second quality score of the negative sample as the weight of the second preference probability, the influence of the sample on the model parameter update can be adjusted according to the explicit sample quality score difference. In the calculation of the weighted preference difference loss, high-quality positive samples will make a greater contribution to the loss function due to their higher quality scores, while low-quality negative samples will contribute less to the loss function due to their lower quality scores. As a result, the model will be more inclined to learn the features of high-quality positive samples during the optimization process, while weakening the features of low-quality negative samples, guiding the model to better learn the features of these samples, effectively widening the gap between positive and negative samples.

[0124] In some embodiments, for obtaining the first quality score of the positive sample and the second quality score of the negative sample in the sample data as described in the above embodiment, the quality scores of the positive and negative samples can be automatically evaluated by using a pre-trained preference ranking model PRM. Figure 4As shown, for each positive and negative sample included in the sample data used in this round of training, the PRM can be used to perform quality scoring on the positive and negative samples included in the sample data, respectively, to obtain a first quality score corresponding to the positive sample and a second quality score corresponding to the negative sample. Optionally, the quality score range scored by the PRM can be limited to [-a, a] through a pre-training process, where a is a positive integer, and the first quality score in the same sample data can be trained to be greater than the second quality score to better reflect the quality score difference between the positive and negative samples.

[0125] In the disclosed embodiment, the PRM model can process sample data in batches and automatically output positive and negative sample scores, significantly improving efficiency and reducing subjective errors in manual annotation, making it suitable for rapid evaluation of large-scale data sets. In addition, by further limiting the quality score to a symmetrical value range, numerical instability caused by extreme values ​​is avoided, and the relative quality difference between positive and negative samples is intuitively presented on a unified scale, avoiding the problem of incomparable absolute scores between different samples. The positive sample score is set higher than the negative sample score to directly reflect the difference in sample quality, so that the model can more clearly identify the learning objectives during the optimization process and avoid preference confusion caused by overlapping or inverted quality scores.

[0126] Corresponding to the embodiment of the aforementioned model training method, see Figure 5 As shown, the present application also provides an embodiment of a model training device, the device comprising:

[0127] The quality score acquisition module 501 is used to acquire, when training the model using the direct preference optimization (DPO) training method, a first quality score of the positive sample in the sample data and a second quality score of the negative sample in the sample data, for each input prompt, positive sample, and negative sample included in the sample data used in the current round of training;

[0128] The model prediction acquisition module 502 is configured to acquire a first probability predicted by the model for the positive example and a second probability predicted for the negative example; the first probability represents the probability that the result of the model processing the input prompt is the positive example; the second probability represents the probability that the result of the model processing the input prompt is the negative example;

[0129] a preference difference loss determination module 503 for performing weighted processing on the first probability and the second probability based on the first quality score and the second quality score to determine a preference difference loss corresponding to the sample data;

[0130] The target loss construction module 504 is used to construct the loss of all sample data used in this round of training as the target loss of the model in this round of training based on the preference difference loss corresponding to each sample data, and the supervised fine-tuning SFT loss of the positive samples and the SFT loss of the negative samples in the sample data, so as to adjust the parameters of the model based on the target loss.

[0131] In some embodiments, the target loss construction module is specifically configured to:

[0132] Using the three preset hyperparameters as weights, the preference difference loss, the SFT loss of the positive sample, and the SFT loss of the negative sample corresponding to the sample data are weighted and summed to obtain the weighted loss of the sample data;

[0133] The weighted loss of each sample data used in this round of training is averaged to obtain the loss of all sample data used in this round of training.

[0134] In some embodiments, the preference difference loss determination module includes:

[0135] a preference probability determination module, configured to determine a first preference probability and a second preference probability based on a probability distribution normalization processing result of the first probability and the second probability; the first preference probability is used to reflect the strength of the model's preference for the positive example; the second preference probability is used to reflect the strength of the model's preference for the negative example;

[0136] A weighted processing module is configured to perform weighted processing on the first preference probability and the second preference probability based on the first quality score and the second quality score.

[0137] In some embodiments, the probability distribution normalization process is implemented by normalization; and the preference probability determination module is specifically configured to:

[0138] Normalizing the first probability and the second probability corresponding to the sample data by a softmax function to obtain normalized first probability and second probability;

[0139] The logarithm of the normalized first probability is used as the first preference probability, and the logarithm of the normalized second probability is used as the second preference probability.

[0140] In some embodiments, the weighted processing module is specifically configured to:

[0141] Multiplying a first quality score corresponding to a positive example in the sample data by a first preference probability corresponding to the positive example to obtain a first result;

[0142] Multiplying the second quality score corresponding to the negative sample in the sample data by the second preference probability corresponding to the negative sample to obtain a second result;

[0143] The negative of the sum of the first result and the second result is used as the preference difference loss corresponding to the sample data.

[0144] In some embodiments, the quality score acquisition module is specifically configured to:

[0145] Using the trained preference ranking model PRM, the positive samples and negative samples included in the sample data are respectively scored to obtain a first quality score corresponding to the positive sample and a second quality score corresponding to the negative sample; wherein the quality score range of the PRM score is [-a, a], where a is a positive integer; the first quality score in the same sample data is greater than the second quality score.

[0146] In some embodiments, the apparatus further comprises:

[0147] The negative logarithm result of the first probability corresponding to the positive example is used as the SFT loss of the positive example, and the negative logarithm result of the second probability corresponding to the negative example is used as the SFT loss of the negative example.

[0148] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0149] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0150] The embodiment of the present application also provides an electronic device, the structural diagram of the electronic device is as follows Figure 6 As shown, the electronic device 600 includes at least one processor 601, a memory 602 and a bus 603, and the at least one processor 601 is electrically connected to the memory 602; the memory 602 is configured to store at least one computer-executable instruction, and the processor 601 is configured to execute the at least one computer-executable instruction, thereby executing the steps of any model training method provided in any embodiment or any optional implementation method in this application.

[0151] Furthermore, the processor 601 may be a Field-Programmable Gate Array (FPGA) or other devices with logic processing capabilities, such as a Microcontroller Unit (MCU) or a Central Processing Unit (CPU).

[0152] An embodiment of the present application also provides another readable storage medium storing a computer program, which is used to implement the steps of any model training method provided in any embodiment or any optional implementation method of the present application when executed by a processor.

[0153] The readable storage media provided in the embodiments of the present application include, but are not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the readable storage medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0154] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential sequence to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0155] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of specific embodiments of specific inventions. Certain features described in multiple embodiments within this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of the sub-combination.

[0156] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: When training the model using the direct preference optimization (DPO) training method, for each input prompt, positive example, and negative example included in each sample data used in this round of training, obtain the first quality score of the positive example and the second quality score of the negative example in the sample data; Obtaining a first probability predicted by the model for the positive example and a second probability predicted for the negative example; the first probability represents the probability that a result of the model processing the input prompt is the positive example; the second probability represents the probability that a result of the model processing the input prompt is the negative example; Based on the first quality score and the second quality score, weighting the first probability and the second probability to determine the preference difference loss corresponding to the sample data; According to the preference difference loss corresponding to each sample data, as well as the supervised fine-tuning SFT loss of the positive samples and the SFT loss of the negative samples in the sample data, the loss of all sample data used in this round of training is constructed as the target loss of the model in this round of training, so as to adjust the parameters of the model based on the target loss.

2. The method according to claim 1, characterized in that The loss of all sample data used in this round of training is constructed, including: Using the three preset hyperparameters as weights, the preference difference loss, the SFT loss of the positive sample, and the SFT loss of the negative sample corresponding to the sample data are weighted and summed to obtain the weighted loss of the sample data; The weighted loss of each sample data used in this round of training is averaged to obtain the loss of all sample data used in this round of training.

3. The method according to claim 1, characterized in that The weighting the first probability and the second probability based on the first quality score and the second quality score includes: Based on the probability distribution normalization results of the first probability and the second probability, a first preference probability and a second preference probability are determined; the first preference probability is used to reflect the preference strength of the model for the positive example; the second preference probability is used to reflect the preference strength of the model for the negative example; The first preference probability and the second preference probability are weighted based on the first quality score and the second quality score.

4. The method according to claim 3, characterized in that The probability distribution normalization processing is implemented by normalization; the first preference probability and the second preference probability are determined based on the probability distribution normalization processing results of the first probability and the second probability, including: Normalizing the first probability and the second probability corresponding to the sample data by a softmax function to obtain normalized first probability and second probability; The logarithm of the normalized first probability is used as the first preference probability, and the logarithm of the normalized second probability is used as the second preference probability.

5. The method according to claim 3, characterized in that The weighting process of the first preference probability and the second preference probability includes: Multiplying a first quality score corresponding to a positive example in the sample data by a first preference probability corresponding to the positive example to obtain a first result; Multiplying the second quality score corresponding to the negative sample in the sample data by the second preference probability corresponding to the negative sample to obtain a second result; The negative of the sum of the first result and the second result is used as the preference difference loss corresponding to the sample data.

6. The method according to claim 1, wherein The obtaining of a first quality score of a positive example and a second quality score of a negative example in the sample data includes: Using the trained preference ranking model PRM, quality scores are respectively performed on the positive samples and negative samples included in the sample data to obtain a first quality score corresponding to the positive sample and a second quality score corresponding to the negative sample; The quality score range of the PRM score is [-a, a], where a is a positive integer; the first quality score in the same sample data is greater than the second quality score.

7. The method according to claim 1, characterized in that The method further comprises: The negative logarithm result of the first probability corresponding to the positive example is used as the SFT loss of the positive example, and the negative logarithm result of the second probability corresponding to the negative example is used as the SFT loss of the negative example.

8. A model training device, characterized in that: The device comprises: a quality score acquisition module for acquiring, when training the model using the direct preference optimization (DPO) training method, a first quality score of the positive sample in the sample data and a second quality score of the negative sample in the sample data, for each input prompt, positive sample, and negative sample included in the sample data used in the current round of training; a model prediction acquisition module, configured to acquire a first probability predicted by the model for the positive example and a second probability predicted for the negative example; the first probability representing the probability that the result of the model processing the input prompt is the positive example; and the second probability representing the probability that the result of the model processing the input prompt is the negative example; a preference difference loss determination module, configured to perform weighted processing on the first probability and the second probability based on the first quality score and the second quality score, to determine a preference difference loss corresponding to the sample data; The target loss construction module is used to construct the loss of all sample data used in this round of training as the target loss of the model in this round of training based on the preference difference loss corresponding to each sample data, the supervised fine-tuning SFT loss of the positive samples in the sample data, and the SFT loss of the negative samples, so as to adjust the parameters of the model based on the target loss.

9. An electronic device, characterized in that: include: Memory, processor; The memory is used to store computer programs; The processor is configured to call the computer program to implement the method according to any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images