Reward model training method and device, reward model sorting method and device, equipment and medium

Abstract

The invention discloses a reward model training method and device, a sorting method and device, equipment and a medium, and is applied to the technical field of computers, and the method comprises the steps: carrying out the grouping of a training data set, and obtaining a preset number of training data subsets; training a to-be-trained reward model by using the preset number of training data subsets to obtain a preset number of initial reward models; determining the confidence coefficient of the training data pair in each training data subset by using a preset number of initial reward models, and filtering the training data pair based on the confidence coefficient to obtain a screened training data set; the confidence is the proportion of the sponsored tickets obtained by the training data pair; and training a target initial reward model in the initial reward models based on the screened training data set to obtain a target reward model. In the process of training the reward model, the training data pair is filtered through the confidence coefficient of the training data pair, so that the quality of the training data pair is improved, and the influence on the fitting of the reward model is prevented.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a reward model training method, a sorting method, a device, a equipment and a medium. Background Art

[0002] The training data of the reward model often contains a large amount of noisy data, that is, a large amount of training data has its positive and negative examples reversed by the annotators, and is mixed in a large amount of training data sets and cannot be picked out. This data will seriously affect the fitting of the model during training.

[0003] Therefore, how to denoise the training data, reduce the impact of noisy data, and improve the fitting performance of the reward model is a technical problem that technicians in this field urgently need to solve. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a reward model training method, ranking method, device, equipment and medium, which solve the technical problem that noise data cannot be filtered in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a reward model training method, comprising:

[0006] Grouping the training data set to obtain a preset number of training data subsets;

[0007] Using the preset number of training data subsets to train multiple reward models to be trained, to obtain a preset number of initial reward models;

[0008] Determine the confidence corresponding to the training data pair in each training data subset using the preset number of initial reward models, and filter all the training data pairs based on the confidence to obtain a filtered training data set; the confidence is the proportion of the training data pairs that obtain affirmative votes in the preset number of initial reward models, and the affirmative vote is a vote in favor of the parameter value of the positive example in the training data pair being greater than the parameter value of the negative example;

[0009] The target initial reward model in the initial reward model is trained based on the filtered training data set to obtain a target reward model.

[0010] Optionally, using the preset number of training data subsets to train multiple reward models to be trained to obtain a preset number of initial reward models includes:

[0011] Determine a reward model to be trained corresponding to each training data subset in the preset number of training data subsets;

[0012] Each of the training data subsets is used to train a corresponding reward model to be trained to obtain the preset number of initial reward models.

[0013] Optionally, the confidence corresponding to the training data pairs in each training data subset is determined using the preset number of initial reward models, and all training data pairs are filtered based on the confidence to obtain a filtered training data set, including:

[0014] Determine the parameter value corresponding to the positive example of each training data pair and the parameter value corresponding to the negative example using the preset number of initial reward models;

[0015] Determining the confidence of each of the training data pairs based on the parameter value corresponding to the positive example and the parameter value corresponding to the negative example;

[0016] When the confidence level is greater than a confidence level threshold, determining not to filter the training data pair;

[0017] When the confidence is not greater than the confidence threshold, it is determined to filter the training data pair.

[0018] Optionally, after training the target initial reward model in the initial reward model based on the filtered training data set to obtain the target reward model, the reward model training method further includes:

[0019] Determine a smoothing coefficient based on the confidence corresponding to each training data pair in the screened training data set;

[0020] Dynamically adjust the loss function corresponding to the target reward model based on the smoothing coefficient corresponding to each training data pair;

[0021] The target reward model is trained based on the loss function to obtain a final target reward model.

[0022] Optionally, dynamically adjusting the loss function corresponding to the target reward model based on the smoothing coefficient corresponding to each training data pair includes:

[0023] The ranking loss function is subjected to weighted label smoothing processing based on the smoothing coefficient to obtain the loss function corresponding to the target reward model.

[0024] Optionally, training the target reward model based on the loss function to obtain a final target reward model includes:

[0025] Based on a preset number of initial reward models, a plurality of parameter values ​​corresponding to the positive examples and the negative examples of each training data pair are determined, and based on the plurality of parameter values, an average difference value corresponding to each training data pair is determined; wherein the average difference value is the average of the differences between the positive example parameter value and the negative example parameter value;

[0026] Determining a dynamic adjustment margin coefficient based on the average difference value, and adjusting the loss function according to the dynamic adjustment margin coefficient to obtain an adjusted loss function;

[0027] The target reward model is trained based on the adjusted loss function until the adjusted loss function converges to obtain the final target reward model.

[0028] Optionally, the training data set is grouped to obtain a preset number of training data subsets, including:

[0029] The training data set is grouped to obtain ten training data subsets; wherein each training data subset includes the same number of training data pairs.

[0030] The present application also provides a sorting method, comprising:

[0031] Acquire multiple large model reply texts corresponding to the command text; wherein the large model reply text is the output text generated by the large model according to the received command text;

[0032] Based on each large model reply text, a reply quality parameter is obtained using a target reward model; wherein the target reward model is a model obtained based on the above-mentioned reward model training method;

[0033] Based on the response quality parameter corresponding to each large model response text, each large model response text is sorted to obtain the target sorted large model response text.

[0034] The present application also provides a reward model training device, comprising:

[0035] A grouping module is used to group the training data set to obtain a preset number of training data subsets;

[0036] An initial reward model determination module, used to train a plurality of reward models to be trained using the preset number of training data subsets to obtain a preset number of initial reward models;

[0037] A data filtering module, used to determine the confidence corresponding to the training data pair in each training data subset using the preset number of initial reward models, and filter all the training data pairs based on the confidence to obtain a filtered training data set; the confidence is the proportion of the training data pair that obtains affirmative votes in the preset number of initial reward models, and the affirmative vote is a vote that agrees that the parameter value of the positive example in the training data pair is greater than the parameter value of the negative example;

[0038] A target reward determination module is used to train a target initial reward model in the initial reward model based on the screened training data set to obtain a target reward model.

[0039] The present application also provides a sorting device, comprising:

[0040] A large model reply text acquisition module is used to acquire multiple large model reply texts corresponding to the command text; wherein the large model reply text is the output text generated by the large model according to the received command text;

[0041] A reply quality parameter determination module, used to obtain a reply quality parameter based on each large model reply text using a target reward model; wherein the target reward model is a model obtained based on the above-mentioned reward model training method;

[0042] The sorting module is used to sort each large model reply text based on the reply quality parameter corresponding to each large model reply text to obtain the target sorted large model reply text.

[0043] The present application also provides an electronic device, including:

[0044] Memory for storing computer programs;

[0045] A processor is used to implement the steps of the above-mentioned reward model training method and ranking method when executing the computer program.

[0046] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned reward model training method and sorting method are implemented.

[0047] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned reward model training method and sorting method when executed by a processor.

[0048] It can be seen that the present invention obtains a preset number of training data subsets by grouping the training data set; uses the preset number of training data subsets to train multiple reward models to be trained to obtain a preset number of initial reward models; uses the preset number of initial reward models to determine the preset number of confidences corresponding to the training data pairs in each training data subset, and filters all the training data pairs based on the confidence to obtain the filtered training data set; the confidence is the proportion of the training data pairs that obtain affirmative votes in the preset number of initial reward models, and the affirmative vote is the vote that the parameter value of the positive example in the training data pair is greater than the parameter value of the negative example, and the target initial reward model in the initial reward model is trained based on the filtered training data set to obtain the target reward model. Compared with the current inability to filter the training data pairs that are marked negative, the present application determines the preset number of confidences corresponding to the training data pairs in each training data subset by using the preset number of initial reward models, thereby filtering the training data pairs based on the preset number of confidences, thereby achieving the filtering of the training data that are marked negative, and improving the fitting performance of the reward model.

[0049] In addition, the present invention also provides a reward model training device, equipment and medium, as well as a sorting method, device, equipment and medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0051] Figure 1 A flowchart of a reward model training method provided by an embodiment of the present invention;

[0052] Figure 2 A flowchart of another reward model training method provided by an embodiment of the present invention;

[0053] Figure 3 A flowchart of a reward model training method provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of the structure of a reward model training method provided by an embodiment of the present invention;

[0055] Figure 5 A flowchart of a sorting method provided by an embodiment of the present invention;

[0056] Figure 6A schematic diagram of the structure of a reward model training device provided by an embodiment of the present invention;

[0057] Figure 7 A schematic diagram of the structure of a sorting device provided by an embodiment of the present invention;

[0058] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] Some of the terms that appear in the description of the embodiments of the present application may be interpreted as follows:

[0061] 1. Reward Model (RM): A model used to evaluate and guide the big model. It optimizes the agent by learning preferences for data. Specifically, it gives a score to the big model’s responses, which can be used to score and rank the big model’s responses or for RLHF training of the big model.

[0062] 2. Pair-wise loss (sort loss function): a loss function that trains the reward model by increasing the distance between positive and negative examples in the solution space;

[0063] 3. Label smoothing: a regularization technique that improves the generalization ability of the model by smoothing the label distribution;

[0064] 4. Dynamic margin: A dynamically adjusted margin used to adjust the pair-wise loss to reflect the preference strength of positive and negative examples in the data pairs.

[0065] Please refer to Figure 1 , Figure 1 A flowchart of a reward model training method provided by an embodiment of the present invention. The method may include:

[0066] S101, grouping the training data set to obtain a preset number of training data subsets.

[0067] The executor of this embodiment is an electronic device. The electronic device can be a computer, a mobile phone, etc. This embodiment does not limit the specific number of groups. For example, this embodiment can divide the training data set into multiple training data subsets, and each training data subset includes the same number of training data pairs, such as dividing the training data set into 9 training data subsets, or this embodiment can divide the training data set into 10 training data subsets, or this embodiment can also divide the training data set into 11 training data subsets, and the training data pairs included in each training data subset can be regarded as a group. Therefore, dividing the training data set into multiple training data subsets can also be referred to as dividing the training data set into multiple groups. It should be noted that, after experiments, when the training data set is divided into 10 groups, the comprehensive performance of filtering and rewarding model training is better. This embodiment does not limit the specific training data set, as long as the training data set includes multiple training data pairs, each training data pair includes prompt, chosen, and reject. The training data set of this embodiment may be a cleaned and formatted data set, and the cleaning method of this embodiment may be cluster-based outlier detection to identify and exclude noise data, or the cleaning method may also be a method of deleting missing values. In this embodiment, the number of training data pairs in each training data subset is consistent.

[0068] S102, using a preset number of training data subsets to train multiple reward models to be trained, to obtain a preset number of initial reward models.

[0069] In this embodiment, the reward model (RM) is a model used to evaluate and guide the large model, specifically to give a score to the reply of the large model, which can be used to score and rank the reply of the large model, or for RLHF (Reinforcement Learning from Human Feedback) training of the large model. In this embodiment, a cross-validation method can be used to train multiple reward models to be trained using a preset number of training data subsets to obtain a preset number of initial reward models. In this embodiment, the number of training data subsets is consistent with the number of initial reward models.

[0070] It should be further explained that the above-mentioned training of multiple reward models to be trained using a preset number of training data subsets to obtain a preset number of initial reward models may include: determining the reward model to be trained corresponding to each training data subset in the preset number of training data subsets; and training the corresponding reward model to be trained using each training data subset to obtain a preset number of initial reward models. This embodiment trains the reward models to be trained using different training data subsets, so that multiple reward models with different parameters can be obtained.

[0071] S103, using a preset number of initial reward models to determine the confidence corresponding to the training data pairs in each training data subset, and filtering all the training data pairs based on the confidence to obtain a filtered training data set, where the confidence is the proportion of the training data pairs that obtain affirmative votes in the preset number of initial reward models, and the affirmative vote is a vote in favor of the parameter value of the positive example in the training data pair being greater than the parameter value of the negative example.

[0072] This embodiment uses a preset number of initial reward models to determine the corresponding confidence of the training data pair in each training data subset, which refers to the proportion of affirmative votes obtained by the training data pair in the preset number of initial reward models. An affirmative vote refers to a vote in favor of the parameter value of the positive example in the training data pair being greater than the parameter value of the negative example. For example, there are 10 initial reward models. For data pair A in the first initial reward model, 4 of the 10 initial reward models voted in favor of data pair A, that is, the 4 initial reward models believe that the parameter value of the positive example in data pair A is better than the parameter value of the negative example. Based on the number of affirmative votes of 4 and the number of initial reward models of 10, it can be determined that the confidence of data pair A is 40%. If the confidence threshold is 50%, it is determined that the confidence of data pair A does not reach the confidence threshold. At this time, it is necessary to filter out the data pair A. According to data pair A; for data pair B in the first initial reward model, 8 out of 10 initial reward models voted in favor of data pair B, that is, the 8 initial reward models believe that the parameter value of the positive example in data pair B is better than the parameter value of the negative example. According to the number of votes in favor 8 and the number of initial reward models 10, it can be determined that the confidence of data pair B is 80%. If the confidence threshold is 50%, it is determined that the confidence of data pair B reaches the confidence threshold. At this time, data pair B is not filtered, that is, data B is selected. In this embodiment, all training data pairs in each training data subset are filtered. This embodiment does not limit the order of filtering, and filtering can be performed in the order of training data subsets; or this embodiment can also sort all training data pairs and then filter. In this embodiment, filtering the training data pairs based on confidence means determining whether the confidence is greater than a set confidence threshold, that is, determining whether the number of initial reward models that believe that the parameter value of the positive example is greater than the parameter value of the negative example is greater than a preset number, and if not, determining to filter the training data pair, that is, when determining the confidence, this embodiment evaluates each training data pair based on a model integration voting mechanism to determine the confidence of the training data pair, wherein the model integration voting mechanism refers to integrating the prediction results of multiple models, which can significantly improve the generalization ability, stability and accuracy of the model. This embodiment can also use different model integration strategies, such as random forests or gradient boosting trees, to improve the diversity and robustness of the model.For ease of understanding, assume that there are training data subsets 1 and 2. Training data subset 1 includes data pair 1, data pair 2, data pair 3, data pair 4, and data pair 5. Training data subset 2 includes data pair 6, data pair 7, data pair 8, data pair 9, and data pair 10. The confidences of data pairs 1 to 5 are 10%, 61%, 70%, 40%, and 80%, respectively. The confidences of data pairs 6 to 10 are 20%, 50%, 10%, 70%, and 90%, respectively. If the current confidence threshold is 60%, then the confidences of data pair 1 and data pair 4 in training data subset 1 are lower than the confidence threshold and need to be filtered out. Data pair 2, data pair 3, and data pair 5 are retained. The confidences of data pair 6, data pair 7, and data pair 8 in training data subset 2 are all lower than 60%, so they need to be filtered out. Data pair 9 and data pair 10 are retained.

[0073] It should be further explained that, in order to improve the accuracy of filtering, the above-mentioned use of a preset number of initial reward models to determine the confidence corresponding to the training data pairs in each training data subset, and filtering all training data pairs based on the confidence to obtain the filtered training data set may include:

[0074] S1031, using a preset number of initial reward models to determine a parameter value corresponding to a positive example of each training data pair, and a parameter value corresponding to a negative example;

[0075] S1032, determining the confidence of each training data pair based on the parameter value corresponding to the positive example and the parameter value corresponding to the negative example;

[0076] S1033, when the confidence level is greater than the confidence level threshold, determining not to filter the training data pair;

[0077] S1034: When the confidence level is not greater than the confidence level threshold, determine to filter the training data pair.

[0078] The parameter value in this embodiment can be a scoring value. It can be understood that the reward model can determine the output score based on the input, that is, each reward model can determine a parameter value of a positive example and a parameter value of a negative example, compare the parameter value of the positive example with the parameter value of the negative example, count the number of initial reward models that agree that the parameter value of the positive example is greater than the parameter value of the negative example, and determine the confidence based on the number. For example, there are 10 initial reward models, of which 5 reward models believe that the parameter value of the positive example is greater than the parameter value of the negative example, and the confidence is 50% at this time. This embodiment sets a confidence threshold, and only training data pairs greater than the confidence threshold will be retained, otherwise they will be filtered, thereby improving the accuracy of filtering training data pairs.

[0079] S104, training a target initial reward model in the initial reward model based on the screened training data set to obtain a target reward model.

[0080] The model trained in this embodiment may be a partial model in the initial reward model; or the model trained in this embodiment may be all models in the initial reward model. This embodiment trains the target initial reward model in the initial reward model based on the screened training data set, and obtaining the target reward model may mean training any initial reward model based on the screened training data set, or this embodiment may also train a preset number of initial reward models based on the screened training data set to obtain a preset number of target reward models. This embodiment does not limit the number of target initial reward models. For example, the target initial reward model in this embodiment may be any one model in the initial reward model; or the target initial reward model in this embodiment may also be a partial model in the initial reward model; or the target initial reward model in this embodiment may also be all models in the initial reward model.

[0081] A reward model learning method provided by an embodiment of the present invention may include: S101, grouping a training data set to obtain a preset number of training data subsets; S102, using a preset number of training data subsets to train multiple reward models to be trained to obtain a preset number of initial reward models; S103, using a preset number of initial reward models to determine the confidence corresponding to the training data pairs in each training data subset, and filtering all training data pairs based on the confidence to obtain a filtered training data set; the confidence is the proportion of the training data pairs that obtain affirmative votes in the preset number of initial reward models, and the affirmative vote is a vote that agrees that the parameter value of the positive example in the training data pair is greater than the parameter value of the negative example; S104, based on the filtered training data set, training the target initial reward model in the initial reward model to obtain a target reward model. The present application determines the confidence corresponding to the training data pair in each training data subset through the initial reward model, and filters the positive and negative examples with negative labels based on the confidence, thereby excluding the training data pairs with low confidence, improving the quality of the training data, and improving the robustness of the reward model to noisy data.

[0082] Please refer to Figure 2 , Figure 2 A flowchart of another reward model training method provided by an embodiment of the present invention. The method may include:

[0083] S201, grouping the training data set to obtain a preset number of training data subsets.

[0084] S201 to S204 in this embodiment are the above-mentioned S101 to S104, so they will not be described again this time.

[0085] S202, using a preset number of training data subsets to train multiple reward models to be trained, to obtain a preset number of initial reward models.

[0086] This embodiment can introduce a multi-task learning framework (such as a ranking task and a scoring task) for the reward model to be trained, and simultaneously optimize the performance of the model on multiple related tasks to improve the generalization ability and application scope of the model.

[0087] S203, using a preset number of initial reward models to determine the confidence corresponding to the training data pairs in each training data subset, and filtering all the training data pairs based on the confidence to obtain a filtered training data set.

[0088] The confidence level in this embodiment is the proportion of training data pairs that obtain affirmative votes in a preset number of initial reward models, where an affirmative vote is a vote in favor of the parameter value of the positive example in the training data pair being greater than the parameter value of the negative example.

[0089] S204: Train the initial reward model based on the screened training data set to obtain a target reward model.

[0090] S205: Determine a smoothing coefficient based on the confidence level corresponding to each training data pair in the screened training data set.

[0091] This embodiment does not limit the specific method for determining the smoothing coefficient based on the confidence corresponding to each training data pair in the screened training data set. For example, when the confidence of the current training data pair is 95%, a smoothing coefficient of 0.95 can be assigned; or this embodiment can dynamically adjust the weight according to the confidence of the model prediction, giving a higher weight to the prediction with high confidence, and vice versa; or this embodiment can also combine other features to weight the confidence to determine the smoothing coefficient. The range of the smoothing coefficient in this embodiment can be a parameter greater than or equal to 0 and less than or equal to 1.

[0092] S206, dynamically adjusting the loss function corresponding to the target reward model based on the smoothing coefficient corresponding to each training data pair.

[0093] This embodiment dynamically adjusts the loss function corresponding to the target reward model based on the smoothing coefficient, which means that each training data pair corresponds to a smoothing coefficient, and it is necessary to use multiple training data pairs and smoothing coefficients to adjust the loss function.

[0094] It should be further explained that the above-mentioned dynamic adjustment of the loss function corresponding to the target reward model based on the smoothing coefficient corresponding to each training data pair may include: performing weighted label smoothing processing on the sorting loss function based on the smoothing coefficient to obtain the loss function corresponding to the target reward model. The Pair-wise loss (sorting loss function) in this embodiment is a loss function that trains the reward model by increasing the distance between positive examples and negative examples in the solution space. Label smoothing in this embodiment is a regularization technique that improves the generalization ability of the model by smoothing the label distribution. The loss function for label smoothing in this embodiment can be ,in is the probability that the model predicts that a positive example is better than a negative example. This is the response for chosen (positive example). That is the response of reject (negative example); is the smoothing parameter, controlling the strength of label smoothing, 0 ≤ < 1, for example, the confidence level is greater than 0%, and the data below 100% accounts for 5%, then = 0.95.

[0095] S207, training the target reward model based on the loss function to obtain a final target reward model.

[0096] When the loss function is obtained in this embodiment, the target reward model can be trained based on the loss function to obtain the final target reward model.

[0097] It should be further explained that, in order to improve the prediction performance of the reward model, the target reward model is trained based on the loss function to obtain the final target reward model, which may include:

[0098] S2071, determining an average difference value corresponding to each training data pair based on a plurality of parameter values ​​corresponding to the positive example and the negative example of each training data pair determined by a preset number of initial reward models; wherein the average difference value is the average of the differences between the positive example parameter value and the negative example parameter value;

[0099] S2072, determining a dynamic adjustment margin coefficient based on the average difference value, and adjusting the loss function according to the dynamic adjustment margin coefficient to obtain an adjusted loss function;

[0100] S2073, training the target reward model based on the adjusted loss function until the adjusted loss function converges to obtain the final target reward model.

[0101] In this embodiment, the average difference value of each training data pair (i.e., the average difference value of the scores of the ten models for positive and negative examples) is used to dynamically adjust the margin in the loss function. This dynamic margin reflects the preference strength of the positive and negative examples in the data pair, so that the model can adjust the score difference between the positive and negative examples accordingly according to the difference in preference strength. In this way, the model can learn human preferences more accurately and improve the prediction accuracy module input of the reward model. The formula is as follows: ; ;in, In order to dynamically adjust the marginal coefficient, combined with the dynamic parameters based on the smoothing coefficient obtained previously, the formula of the final loss function is . The dynamic margin model is added to predict the probability that the positive example is better than the negative example.

[0102] The reward model training method provided by the embodiment of the present invention may include: S201, grouping the training data set to obtain a preset number of training data subsets. S202, using a preset number of training data subsets to train multiple reward models to be trained to obtain a preset number of initial reward models. S203, using a preset number of initial reward models to determine the confidence corresponding to the training data pairs in each training data subset, and filtering all training data pairs based on the confidence to obtain a filtered training data set. S204, training the initial reward model based on the filtered training data set to obtain a target reward model. S205, determining a smoothing coefficient based on the confidence corresponding to each training data pair in the filtered training data set. S206, dynamically adjusting the loss function corresponding to the target reward model based on the smoothing coefficient corresponding to each training data pair. S207, training the target reward model based on the loss function to obtain the final target reward model. In the process of training the reward model, the present application improves the quality of training data by identifying and excluding training data pairs with low confidence, and enhances the robustness of the model to noisy data based on the smoothing mechanism of data with questionable confidence, and dynamically adjusts the score gap by dynamically adjusting the marginal coefficient according to the preference strength of the data pair, thereby improving the learning and prediction performance of the reward model.

[0103] In order to make the present invention easier to understand, please refer to Figure 3 , Figure 3 A flowchart of a reward model training method provided by an embodiment of the present invention may specifically include:

[0104] S301, obtaining a training data set, and evenly dividing the training data set into ten training data subsets.

[0105] A structural diagram of a reward model training method corresponding to this embodiment is as follows Figure 4 As shown, Figure 4 The RLHF-Dataset in is a training data set, Data_1 represents the first training data subset, Data_2 represents the second training data subset, Data_3 represents the third training data subset, and Data_4 represents the fourth training data subset, that is, the number after "Data_" represents the number of the training data subset. This embodiment can clean and format the original training data set to ensure that the data pair format is unified as {prompt; chosen; reject} to obtain the training data set.

[0106] S302, using ten training data subsets to train the reward model to be trained, to obtain ten initial reward models; wherein each training data subset corresponds to a reward model to be trained.

[0107] The ten initial reward models in this embodiment correspond to Figure 4 Ten Reward Models in .

[0108] S303, based on the ten initial reward models, the training data pairs in the training data subset are processed using the model integration voting mechanism to determine the parameter values ​​of ten positive examples and the parameter values ​​of negative examples corresponding to each training data pair; the training data pairs include command text, positive examples corresponding to the command text, and negative examples corresponding to the command text.

[0109] S304 , determining the confidence of each training data pair based on the parameter values ​​of the ten positive examples and the parameter values ​​of the negative examples.

[0110] S305, when the confidence of the current training data pair is lower than the preset confidence threshold, the current training data is determined to be noise data, and filtered until all training data pairs are filtered and judged to obtain the target training data set.

[0111] The noise data in this embodiment corresponds to Figure 4 The Untrust data in the filter is the data that does not meet the confidence requirements. Figure 4 The Filtered Data in the example represents the filtering of the training data. Figure 4 This embodiment uses a ten-fold cross-validation and model voting mechanism to identify and exclude noise data with low confidence, thereby improving the quality of training data.

[0112] S306: Determine a smoothing coefficient corresponding to each training data pair according to the confidence level corresponding to each training data pair in the target training data set.

[0113] The smoothing coefficient in this embodiment corresponds to Figure 4For data pairs whose confidence level does not reach 100%, the smoothing coefficient is calculated based on their proportion in the data set.

[0114] S307, performing weighted label smoothing processing on the sorting loss function based on the smoothing coefficient to obtain a loss function to be adjusted corresponding to the reward model.

[0115] This embodiment determines a smoothing coefficient according to the confidence level, adds label smoothing to the loss function, and adjusts the confidence level of the model output.

[0116] S308 , determining an average difference value corresponding to each training data pair in the target training data set based on the parameter values ​​of the ten positive examples and the parameter values ​​of the negative examples.

[0117] S309, determining a dynamic adjustment margin coefficient based on the average difference value, and adjusting the loss function to be adjusted according to the dynamic adjustment margin coefficient to obtain an adjusted loss function.

[0118] The dynamically adjusted marginal coefficient in this embodiment corresponds to Figure 4 The adjusted loss function in this embodiment corresponds to Figure 4 A Novel Loss Func in the example represents a new loss function. The target training data in this example corresponds to Figure 4 New data in the new data set. This embodiment dynamically adds Dynamic margin to adjust the loss function of each pair of data according to the result of model integration to reflect the preference strength of different data pairs. The dynamic adjustment margin in this embodiment can make the ones that should be pulled larger and the ones with small differences that should not be pulled larger, so that the output result of the reward model is more accurate. That is, the technical problem that this embodiment solves by dynamically adjusting the marginal coefficient is the problem of unbalanced preference strength: the preference strength of each pair of data also has strong and weak points. Since the positive and negative examples only need to mark which one is better, that is, qualitative rather than quantitative, and some data positive examples are of much higher quality than the negative examples, the gap between the positive and negative examples should be pulled larger. On the contrary, some positive examples are only a little better than the negative examples, so there is no need to open such a large gap.

[0119] S310, training the initial reward model based on the adjusted loss function until the adjusted loss function converges to obtain a target reward model.

[0120] The target reward model in this embodiment corresponds to Figure 4The target reward model in this embodiment can be applied to provide the ranking of reply results for the dialogue scenario, score the ranking results of the large model and then return the results, that is, the backend service of artificial intelligence virtual human dialogue products. And the subsequent RLHF training (Reinforcement Learning from Human Feedback, reinforcement learning training method based on human feedback).

[0121] The embodiment of the present invention effectively eliminates the influence of noise data by filtering training data pairs with low confidence, thereby improving the robustness of the model in the face of imperfect data. This embodiment uses a weighted labeling method of a smoothing coefficient to enable the model to express the preference strength of different data pairs in a more detailed manner, thereby improving the expressiveness and adaptability of the model. Through label smoothing and dynamic adjustment of margins, the model pays more attention to generalization ability during training and reduces the risk of overfitting. By dynamically adjusting the pair-wise loss, the model training process is more flexible and can be adaptively optimized according to data characteristics, thereby improving the learning and prediction performance of the target reward model.

[0122] Please refer to Figure 5 , Figure 5 A flowchart of a sorting method provided by an embodiment of the present invention. The method may include:

[0123] S501, obtaining multiple large model reply texts corresponding to the command text; wherein the large model reply text is the output text generated by the large model according to the received command text.

[0124] The command text of this embodiment is the input of the large model, for example, a question, or a dialogue script.

[0125] S502, based on each large model reply text, using the target reward model to obtain the reply quality parameter; wherein the target reward model is a model obtained based on the above-mentioned reward model training method.

[0126] S503, sorting each large model reply text based on the reply quality parameter corresponding to each large model reply text to obtain the target sorted large model reply text.

[0127] This embodiment can sort the large model reply texts based on the reply quality parameters obtained by one target reward model, and can also sort the large model reply texts based on the reply quality parameters obtained by multiple target reward models.

[0128] The embodiment of the present invention uses the target reward model provided above when performing sorting. Since the target reward model has a higher accuracy in determining the response quality parameter, the accuracy of sorting can be improved.

[0129] The reward model training device provided in an embodiment of the present invention is introduced below. The reward model training device described below and the reward model training method described above can be referenced to each other.

[0130] Please refer to Figure 6 , Figure 6 A schematic diagram of a structure of a reward model training device provided by an embodiment of the present invention may include:

[0131] A grouping module 100 is used to group the training data set to obtain a preset number of training data subsets;

[0132] An initial reward model determination module 200 is used to train a plurality of reward models to be trained using the preset number of training data subsets to obtain a preset number of initial reward models;

[0133] The data filtering module 300 is used to determine the confidence corresponding to the training data pair in each training data subset using the preset number of initial reward models, and filter all the training data pairs based on the confidence to obtain a filtered training data set; the confidence is the proportion of the training data pair that obtains affirmative votes in the preset number of initial reward models, and the affirmative vote is a vote that agrees that the parameter value of the positive example in the training data pair is greater than the parameter value of the negative example;

[0134] The target reward determination module 400 is used to train the target initial reward model in the initial reward model based on the screened training data set to obtain a target reward model.

[0135] Further, based on the above embodiment, the initial reward model determination module 200 may include:

[0136] A corresponding relationship determination unit, used to determine the reward model to be trained corresponding to each training data subset in the preset number of training data subsets;

[0137] A training unit is used to train the corresponding reward model to be trained using each of the training data subsets to obtain the preset number of initial reward models.

[0138] Further, based on any of the above embodiments, the data filtering module 300 may include:

[0139] A parameter value determination unit, used to determine the parameter value corresponding to the positive example of each training data pair and the parameter value corresponding to the negative example by using the preset number of initial reward models;

[0140] A confidence determination unit, configured to determine the confidence of each of the training data pairs based on the parameter value corresponding to the positive example and the parameter value corresponding to the negative example;

[0141] a non-filtering determination unit, configured to determine not to filter the training data pair when the confidence level is greater than a confidence level threshold;

[0142] A filtering determination unit is used to determine to filter the training data pair when the confidence is not greater than the confidence threshold.

[0143] Further, based on any of the above embodiments, the reward model training device may further include:

[0144] A smoothing coefficient determination module, used to determine a smoothing coefficient based on the confidence level corresponding to each training data pair in the screened training data set;

[0145] A dynamic adjustment module, used to dynamically adjust the loss function corresponding to the target reward model based on the smoothing coefficient corresponding to each training data pair;

[0146] The final target reward model determination module is used to train the target reward model based on the loss function to obtain the final target reward model.

[0147] Further, based on the above embodiment, the dynamic adjustment module may include:

[0148] A weighted label smoothing processing unit is used to perform weighted label smoothing processing on the sorting loss function based on the smoothing coefficient to obtain the loss function corresponding to the target reward model.

[0149] Further, based on the above embodiment, the final target reward model determination module may include:

[0150] an average difference value determining unit, configured to determine an average difference value corresponding to each training data pair based on a plurality of parameter values ​​corresponding to the positive example and the negative example of each training data pair determined by a preset number of initial reward models, based on the plurality of parameter values; wherein the average difference value is an average of the differences between the positive example parameter value and the negative example parameter value;

[0151] A dynamic adjustment margin coefficient determination unit, configured to determine a dynamic adjustment margin coefficient based on the average difference value, and adjust the loss function according to the dynamic adjustment margin coefficient to obtain an adjusted loss function;

[0152] The final target reward model determination unit is used to train the target reward model based on the adjusted loss function until the adjusted loss function converges to obtain the final target reward model.

[0153] Further, based on the above embodiment, the grouping module 100 may include:

[0154] The grouping module is used to group the training data set to obtain ten training data subsets; wherein each training data subset includes the same number of training data pairs.

[0155] It should be noted that the order of the modules and units in the above-mentioned reward model training device can be changed without affecting the logic.

[0156] The reward model training device provided by an embodiment of the present invention may include: a grouping module 100, which is used to group the training data set to obtain a preset number of training data subsets; an initial reward model determination module 200, which is used to train multiple reward models to be trained using the preset number of training data subsets to obtain a preset number of initial reward models; a data filtering module 300, which is used to determine the confidence corresponding to the training data pair in each training data subset using the preset number of initial reward models, and filter all training data pairs based on the confidence to obtain a screened training data set; the confidence is the proportion of training data pairs that obtain affirmative votes in the preset number of initial reward models, and the affirmative vote is a vote in favor of the parameter value of the positive example in the training data pair being greater than the parameter value of the negative example; the confidence is the proportion of the parameter value of the positive example being greater than the parameter value of the negative example; a target reward determination module 400, which is used to train the target initial reward model in the initial reward model based on the screened training data set to obtain a target reward model. The present application determines the confidence corresponding to the training data pairs in each training data subset through an initial reward model, and filters the negatively labeled positive and negative examples based on the confidence, thereby excluding training data pairs with low confidence, improving the quality of the training data, and improving the robustness of the reward model to noisy data.

[0157] The following is an introduction to a sorting device provided in an embodiment of the present invention. The sorting device described below and the sorting method described above can be referenced to each other.

[0158] Please refer to Figure 7 , Figure 7 A schematic diagram of a structure of a sorting device provided in an embodiment of the present invention may include:

[0159] The large model reply text acquisition module 500 is used to acquire a plurality of large model reply texts corresponding to the command text; wherein the large model reply text is the output text generated by the large model according to the received command text;

[0160] A reply quality parameter determination module 600 is used to obtain a reply quality parameter based on each large model reply text using a target reward model; wherein the target reward model is a model obtained based on the above-mentioned reward model training method;

[0161] The sorting module 700 is used to sort each large model reply text based on the reply quality parameter corresponding to each large model reply text to obtain a target sorted large model reply text.

[0162] It should be noted that the order of the modules and units in the above-mentioned sorting device can be changed before or after without affecting the logic.

[0163] An electronic device provided by an embodiment of the present invention is introduced below. The electronic device described below and the reward model training method and ranking method described above can be referenced to each other.

[0164] Please refer to Figure 8 , Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention may include:

[0165] A memory 10, used for storing computer programs;

[0166] The processor 20 is used to execute a computer program to implement the above-mentioned reward model training method and ranking method.

[0167] The memory 10 , the processor 20 , and the communication interface 30 all communicate with each other via a communication bus 40 .

[0168] In the embodiment of the present invention, the memory 10 is used to store one or more programs, and the program may include program code, and the program code includes computer operation instructions. In the embodiment of the present invention, the memory 10 may store programs for implementing the following functions:

[0169] Grouping the training data set to obtain a preset number of training data subsets;

[0170] Using a preset number of training data subsets to train multiple reward models to be trained, to obtain a preset number of initial reward models;

[0171] The confidence corresponding to the training data pair in each training data subset is determined using a preset number of initial reward models, and all training data pairs are filtered based on the confidence to obtain a filtered training data set; the confidence is the proportion of the training data pair that obtains affirmative votes in the preset number of initial reward models, and the affirmative vote is a vote that agrees that the parameter value of the positive example in the training data pair is greater than the parameter value of the negative example;

[0172] The target initial reward model in the initial reward model is trained based on the filtered training data set to obtain a target reward model.

[0173] Alternatively, the memory 10 may store a program for implementing the following functions:

[0174] Acquire multiple large model reply texts corresponding to the command text; wherein the large model reply text is the output text generated by the large model according to the received command text;

[0175] Based on each large model reply text, a reply quality parameter is obtained using a target reward model; wherein the target reward model is a model obtained based on the above-mentioned reward model training method;

[0176] Based on the response quality parameters corresponding to each large model response text, each large model response text is sorted to obtain the target sorted large model response text.

[0177] In a possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0178] In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include an NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0179] The processor 20 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, a microprocessor or any conventional processor, etc. The processor 20 may call a program stored in the memory 10 .

[0180] The communication interface 30 may be an interface of a communication module, and is used to connect to other devices or systems.

[0181] Of course, it should be noted that Figure 8 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present invention. In actual applications, the electronic device may include Figure 8 More or fewer components than shown, or combinations of certain components.

[0182] The computer-readable storage medium provided in an embodiment of the present invention is introduced below. The computer-readable storage medium described below and the reward model training method and ranking method described above can be referenced to each other.

[0183] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned reward model training method and sorting method are implemented.

[0184] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0185] Based on this, an embodiment of the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned reward model training method and sorting method when executed by a processor.

[0186] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0187] The professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0188] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0189] The above is a detailed introduction to the reward model training method, sorting method, device, equipment and medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A reward model training method, characterized in that: include: Grouping the training data set to obtain a preset number of training data subsets; Using the preset number of training data subsets to train multiple reward models to be trained, to obtain a preset number of initial reward models; Determine the confidence corresponding to the training data pair in each training data subset using the preset number of initial reward models, and filter all the training data pairs based on the confidence to obtain a filtered training data set; the confidence is the proportion of the training data pairs that obtain affirmative votes in the preset number of initial reward models, and the affirmative vote is a vote in favor of the parameter value of the positive example in the training data pair being greater than the parameter value of the negative example; The target initial reward model in the initial reward model is trained based on the filtered training data set to obtain a target reward model.

2. The reward model training method according to claim 1, characterized in that: Using the preset number of training data subsets to train multiple reward models to be trained to obtain a preset number of initial reward models, including: Determine a reward model to be trained corresponding to each training data subset in the preset number of training data subsets; Each of the training data subsets is used to train a corresponding reward model to be trained to obtain the preset number of initial reward models.

3. The reward model training method according to claim 1, characterized in that: Determine the confidence corresponding to the training data pairs in each training data subset using the preset number of initial reward models, and filter all the training data pairs based on the confidence to obtain a filtered training data set, including: Determine the parameter value corresponding to the positive example of each training data pair and the parameter value corresponding to the negative example using the preset number of initial reward models; Determining the confidence of each of the training data pairs based on the parameter value corresponding to the positive example and the parameter value corresponding to the negative example; When the confidence level is greater than a confidence level threshold, determining not to filter the training data pair; When the confidence is not greater than the confidence threshold, it is determined to filter the training data pair.

4. The reward model training method according to any one of claims 1 to 3, characterized in that: After training the target initial reward model in the initial reward model based on the screened training data set to obtain the target reward model, the reward model training method further includes: Determine a smoothing coefficient based on the confidence corresponding to each training data pair in the screened training data set; Dynamically adjust the loss function corresponding to the target reward model based on the smoothing coefficient corresponding to each training data pair; The target reward model is trained based on the loss function to obtain a final target reward model.

5. The reward model training method according to claim 4, characterized in that: Based on the smoothing coefficient corresponding to each training data pair, dynamically adjusting the loss function corresponding to the target reward model includes: The ranking loss function is subjected to weighted label smoothing processing based on the smoothing coefficient to obtain the loss function corresponding to the target reward model.

6. The reward model training method according to claim 4, characterized in that: The target reward model is trained based on the loss function to obtain a final target reward model, including: Based on a preset number of initial reward models, a plurality of parameter values ​​corresponding to the positive examples and the negative examples of each training data pair are determined, and based on the plurality of parameter values, an average difference value corresponding to each training data pair is determined; wherein the average difference value is the average of the differences between the positive example parameter value and the negative example parameter value; Determining a dynamic adjustment margin coefficient based on the average difference value, and adjusting the loss function according to the dynamic adjustment margin coefficient to obtain an adjusted loss function; The target reward model is trained based on the adjusted loss function until the adjusted loss function converges to obtain the final target reward model.

7. The reward model training method according to claim 1, characterized in that: The training data set is grouped to obtain a preset number of training data subsets, including: The training data set is grouped to obtain ten training data subsets; wherein each training data subset includes the same number of training data pairs.

8. A sorting method, characterized in that: include: Acquire multiple large model reply texts corresponding to the command text; wherein the large model reply text is the output text generated by the large model according to the received command text; Based on each large model reply text, a reply quality parameter is obtained using a target reward model; wherein the target reward model is a model obtained based on the reward model training method according to any one of claims 1 to 7; Based on the response quality parameter corresponding to each large model response text, each large model response text is sorted to obtain the target sorted large model response text.

9. A reward model training device, characterized in that: include: A grouping module is used to group the training data set to obtain a preset number of training data subsets; An initial reward model determination module, used to train a plurality of reward models to be trained using the preset number of training data subsets to obtain a preset number of initial reward models; A data filtering module, used to determine the confidence corresponding to the training data pair in each training data subset using the preset number of initial reward models, and filter all the training data pairs based on the confidence to obtain a filtered training data set; the confidence is the proportion of the training data pair that obtains affirmative votes in the preset number of initial reward models, and the affirmative vote is a vote that agrees that the parameter value of the positive example in the training data pair is greater than the parameter value of the negative example; A target reward determination module is used to train a target initial reward model in the initial reward model based on the screened training data set to obtain a target reward model.

10. A sorting device, characterized in that: include: A large model reply text acquisition module is used to acquire multiple large model reply texts corresponding to the command text; wherein the large model reply text is the output text generated by the large model according to the received command text; A reply quality parameter determination module, used to obtain a reply quality parameter based on each large model reply text using a target reward model; wherein the target reward model is a model obtained based on the reward model training method according to any one of claims 1 to 7; The sorting module is used to sort each large model reply text based on the reply quality parameter corresponding to each large model reply text to obtain the target sorted large model reply text.

11. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method according to any one of claims 1 to 8 when executing the computer program.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

13. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned reward model training method and sorting method.

Images