A training method and device of a reward model based on inter-group contrast

Abstract

This invention provides a training method and apparatus for a reward model based on inter-group comparison, relating to the field of model training technology. The method includes: constructing a training sample set; the training sample set includes a preset target question and multiple sets of training samples corresponding to the target question, each set of training samples including multiple answers labeled with different quality groups; inputting the target question and each answer into an original reward model, outputting a first reward corresponding to each answer; determining a reward signal corresponding to each first reward based on the second rewards corresponding to multiple target answers in the same quality group and a preset reward rule; adjusting the parameters of the original reward model based on the reward signals to obtain the reward model. This implementation compares data in groups, avoiding the computational bottleneck of traditional pairwise reward generation models. It retains the architecture of single-point reward generation models and avoids noise problems caused by subjective differences.

Description

Technical Field

[0001] This invention relates to the field of model training technology, and in particular to a training method and apparatus for a reward model based on inter-group comparison. Background Technology

[0002] Reward models are a core component of reinforcement learning training. Their goal is to learn human preferences for model-generated results, providing precise feedback signals for model optimization. Generative Reward Models (GRMs) can be further divided into single-point and pairwise GRMs based on their scoring methods. Single-point GRMs take question-answer pairs (a question and an answer) as input and output the reward (absolute score) for that answer to evaluate a single response. This scoring method is the simplest, most stable, and fastest inference method. However, its output depends on the labeled scores of each training sample in the training set. Different annotators may label the same training sample differently due to subjective evaluations, resulting in significant noise and making it unsuitable as an effective supervisory signal. Pairwise GRMs compare every two training samples in the training set, focusing only on relative preferences and not absolute scores. The problem with this scoring method is that the evaluation complexity increases exponentially with the number of training samples (n), becoming unbearable in multiple iterations of reinforcement learning training. This significantly limits the application scenarios for large-scale training and the efficiency of online iteration. Therefore, there is an urgent need for a training method for reward models that can integrate the idea of ​​pairwise comparison into single-point generative reward models. This would allow the trained reward model to retain the advantages of single-point generative reward models while simultaneously preserving the supervision signals of pairwise preferences, and would not suffer from the computational bottleneck of traditional pairwise generative reward models. Summary of the Invention

[0003] This invention provides a training method and apparatus for a reward model based on inter-group comparison. By labeling answers with quality groups during the construction of the training sample set, multiple answers corresponding to the target question are grouped according to quality. This allows the subsequent generation of reward signals based on the quality groups labeled with the answers and preset reward rules to only require comparison at the group level, avoiding the computational bottleneck of traditional pairwise reward generation models. Simultaneously, it still uses the question-answer pair format of the target question and answers as input, preserving the architecture of a single-point reward generation model. This allows the trained reward model to output rewards based on individual answers, avoiding the noise problem caused by subjective differences in traditional single-point reward generation models.

[0004] This invention provides a training method and apparatus for a reward model based on inter-group comparison, comprising the following steps: constructing a training sample set; wherein the training sample set includes a preset target question and multiple sets of training samples corresponding to the target question, each set of training samples including multiple answers labeled with different quality groups; inputting the target question and each answer into an original reward model, and outputting a first reward corresponding to each answer; determining a reward signal corresponding to each first reward based on the second rewards corresponding to multiple target answers corresponding to the same quality group in the answer and a preset reward rule; adjusting the parameters of the original reward model according to the reward signal to obtain a reward model.

[0005] Optionally, constructing training samples includes: generating prompt words based on a preset target question; the prompt words indicating multiple answers of different quality to be generated based on the target question; repeatedly performing the following process to construct multiple sets of training samples corresponding to the target question: inputting the prompt words into a large model and outputting multiple answers corresponding to the target question; and labeling each answer with a quality group.

[0006] Optionally, the step of inputting the target question and each answer into the original reward model and outputting the first reward corresponding to each answer includes: inputting the target question and each answer into the original reward model respectively, generating the reasoning text corresponding to each answer using a joint generation method; and obtaining the single-point first reward corresponding to each answer based on the reasoning text.

[0007] Optionally, the quality grouping includes a first quality group indicating excellent quality and a second quality group indicating poor quality; determining the reward signal corresponding to each first reward based on the second rewards corresponding to multiple target answers corresponding to the same quality group and a preset reward rule includes: determining multiple target first answers corresponding to the first quality group and multiple target second answers corresponding to the second quality group from the multiple answers; calculating the first statistical score corresponding to the multiple target first answers and the second statistical score corresponding to the multiple target second answers respectively; for each answer: determining the reward signal corresponding to the answer based on the first reward, the first statistical score, the second statistical score and the preset reward rule.

[0008] Optionally, the first statistical score is calculated in any of the following ways: taking the quantile of the target first reward corresponding to the plurality of target first answers as the first statistical score; taking the mean of the target first reward corresponding to the plurality of target first answers as the first statistical score; taking the ranking quality corresponding to the plurality of target first answers as the first statistical score.

[0009] Optionally, the second statistical score is calculated in any of the following ways: taking the quantile of the target second reward corresponding to the plurality of target second answers as the second statistical score; taking the mean of the target second reward corresponding to the plurality of target second answers as the second statistical score; taking the ranking quality corresponding to the plurality of target second answers as the second statistical score.

[0010] Optionally, for each answer, determining the reward signal corresponding to the answer based on the first reward, the first statistical score, the second statistical score, and a preset reward rule includes: for each target first answer, obtaining the first reward signal corresponding to the target first answer based on the relationship between the target first reward and the second statistical score; for each target second answer, obtaining the second reward signal corresponding to the target second answer based on the relationship between the target second reward and the first statistical score.

[0011] Optionally, obtaining the first reward signal corresponding to the first target answer based on the relationship between the first target reward and the second statistical score includes: If the target first reward is greater than the second statistical score, the first reward signal corresponding to the target first answer is set as a first threshold. If the target first reward is less than or equal to the second statistical score, the first reward signal corresponding to the target first answer is set as the second threshold. And / or, The step of obtaining the second reward signal corresponding to the second target answer based on the relationship between the second target reward and the first statistical score includes: If the target second reward is less than the first statistical score, the second reward signal corresponding to the target second answer is set as the third threshold. If the target second reward is greater than or equal to the first statistical score, the second reward signal corresponding to the target second answer is set as the fourth threshold.

[0012] Optionally, the method further includes: obtaining a format reward corresponding to each of the answers based on whether the reasoning text satisfies a preset format rule; The step of adjusting the parameters of the original reward model according to the reward signal to obtain the reward model includes: calculating the total reward based on the first reward signal, the second reward signal, and the formatted reward; and adjusting the parameters of the original reward model according to the total reward to obtain the reward model.

[0013] The present invention also provides a training device for a reward model based on inter-group comparison, comprising the following modules: An acquisition module is used to construct a training sample set; wherein, the training sample set includes a preset target question and multiple sets of training samples corresponding to the target question, and each set of training samples includes multiple answers labeled with different quality groups; The generation module is used to input the target question and each answer into the original reward model, and output the first reward corresponding to each answer; The determination module is used to determine the reward signal corresponding to each of the first rewards based on the second rewards corresponding to multiple target answers that correspond to the same quality group in the answers and the preset reward rules. The adjustment module is used to adjust the parameters of the original reward model according to the reward signal to obtain the reward model.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training method of the reward model based on inter-group comparison as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method of the reward model based on inter-group comparison as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a training method for a reward model based on inter-group comparison as described above.

[0017] The training method and apparatus for a reward model based on inter-group comparison provided by this invention, by labeling answers with quality groups during the construction of the training sample set, groups multiple answers corresponding to the target question according to quality. This allows the subsequent generation of reward signals based on the quality groups labeled with the answers and preset reward rules to only compare at the group level, avoiding the computational bottleneck of traditional pairwise reward generation models. Simultaneously, it still uses the question-answer pair format of the target question and answers as input, preserving the architecture of a single-point reward generation model. This allows the trained reward model to output rewards based on individual answers, and avoids the noise problem caused by subjective differences in traditional single-point reward generation models. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the training method for the reward model based on inter-group comparison provided by this invention.

[0020] Figure 2 This is a schematic diagram of the process for constructing training samples provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the specific process of step 103 provided by the present invention.

[0022] Figure 4 This is the second flowchart illustrating the training method for the reward model based on inter-group comparison provided by this invention.

[0023] Figure 5 This is a schematic diagram of the training device for the reward model based on inter-group comparison provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] Figure 1 This is one of the flowcharts illustrating the training method for the reward model based on inter-group comparison provided by this invention, such as... Figure 1 As shown, the method includes the following: Step 101: Construct a training sample set; wherein, the training sample set includes a preset target question and multiple sets of training samples corresponding to the target question, and each set of training samples includes multiple answers labeled with different quality groups; In this embodiment of the invention, the ultimate goal of training the reward model is to enable the model to output answers that conform to human preferences, such as being more relevant, more accurate, and safer, rather than to obtain an objective reward score. For example, when the target question is "How to make braised pork," human preference is for answers with more detailed steps rather than those with fewer steps. However, the detail and brevity of steps are relative. For answer 1, which includes three steps, it is impossible to assign a reasonable and unique absolute score without other answers. Therefore, in this embodiment of the invention, instead of using the absolute score method of traditional single-point reward generation models, each answer is labeled with a quality group. This allows the reward model to adjust the target reward for each answer during training based on its corresponding quality group and preset reward rules, ensuring that answers labeled with different quality groups are always in the appropriate order.

[0027] In this approach, a single target question may have multiple answers, each with a different quality group. For example, there are two quality groups: a preferred group and a rejected group. If the target question is "How to make braised pork", answer 1 includes a rough outline of 3 steps, while answer 2 includes a detailed outline of 6 steps, then answer 1 will be labeled as the preferred group, and answer 2 will be labeled as the rejected group.

[0028] Specifically, the answers can be constructed manually based on the target question, or they can be generated in batches using existing open-source large models (such as Chatgpt, Doubao, and Wenxin Yiyan). In one optional embodiment, the process of constructing training samples is as follows: Figure 2 As shown, it includes: Step 201: Generate prompt words based on the preset target question; the prompt words indicate the generation of multiple answers of different quality based on the target question; Repeat steps 202 and 203 below until a sufficient number of training samples are obtained: Step 202: Input the prompt words into the large model and output multiple answers corresponding to the target question; Step 203: Label each answer with a quality group.

[0029] To ensure the diversity of training samples, the prompt words can be input into multiple large models, and each input can be repeated multiple times. Although the same prompt words are input each time, each response from the large model is a separate thought process, thus generating multiple different answers. In an optional embodiment, the quality grouping can include a first quality group indicating excellent quality and a second quality group indicating poor quality. For example, using "how to make braised pork" as the target question, the generated prompt words could be "Please give me two ways to make braised pork." By inputting these prompt words into the large model, two answers can be obtained. The two different answers can be distinguished by their wording, logical clarity, and other human preferences. Therefore, the two different answers can be manually labeled as the preferred group and the rejected group, respectively. Understandably, each time a prompt word is input, two different answers will be labeled as the preferred group A and the rejected group B. Therefore, after repeating steps 202 and 203 multiple times, both the preferred and rejected groups will contain multiple answers, resulting in preferred group A = {a1,a2,...,an} and rejected group B = {b1,b2,...,bn}. It is evident that the training samples constructed through steps 201 to 203 actually include the target question X, preferred group A (n answers), and rejected group B (n answers). Understandably, the quality groups can also be divided into two or more groups, such as three groups: a first quality group indicating excellent quality, a second quality group indicating moderate quality, and a third quality group indicating poor quality. Compared to dividing into two groups, this approach, by extending the inference time, efficiently and accurately compares the quality of answers across multiple groups, significantly improving the efficiency and accuracy of evaluating answer quality between groups.

[0030] Step 102: Input the target question and each answer into the original reward model, and output the first reward corresponding to each answer; In this invention, each answer is actually generated based on the target question, thus the target question and each answer can form a separate question-answer pair. In this embodiment, using traditional question-answer pairs as input to the model allows for the output of reward scores that align with human preferences based on a single answer during subsequent use of the reward model, preserving the superior performance of a single-point generation reward model.

[0031] In practical applications, generative reward models can be categorized not only based on different scoring methods but also based on different generation methods, such as joint generation, distributed generation, and conditional generation. The scoring method determines how scores are assigned, while the generation method determines how they are generated. In an optional embodiment of the present invention, step 102 specifically includes: inputting the target question and each answer separately into the original reward model; using a joint generation method to generate the inference text corresponding to each answer; and obtaining a single-point first reward for each answer based on the inference text. The inference text can be obtained based on multiple preset evaluation dimensions, such as relevance, instruction compliance, and factuality. After setting the preset dimensions, the original reward model can evaluate according to the dimensions and obtain the inference text.

[0032] It should be noted that in this step, the scoring of each answer is relatively independent, and there is no deliberate bias based on the different quality groups labeled for the answers, thus ensuring the independence of each answer's score. Moreover, the joint generation method can overcome the weakness of traditional single-point generation reward models in terms of interpretability.

[0033] Step 103: Based on the second rewards corresponding to multiple target answers that correspond to the same quality group in the answers and the preset reward rules, determine the reward signal corresponding to each first reward; The target answer refers to the answer corresponding to the same quality group, and the second reward refers to the reward corresponding to the target answer, which is essentially the same as the first reward corresponding to the answer. The terms "first" and "second" are only used to distinguish the different answers they correspond to and do not constitute any other limiting function.

[0034] This step is crucial in allocating reward signals in this embodiment of the invention, and it is also the key step that enables the invention to train the model based on human preferences. After step 102, the first reward corresponding to each answer is obtained, but essentially the model parameters have not been adjusted, and the original reward model has not yet been trained. In this step, the second reward corresponding to the target answer in the quality group can be used as a reference to assign a reinforcement learning reward signal to each first reward in step 102. This not only guides the reward model to output higher or lower rewards in a relative preference manner, but also avoids the computational problem of traditional pairwise reward generation models growing exponentially with the number of samples by comparing only the quality group.

[0035] In an optional embodiment, when the quality grouping includes a first quality group indicating excellent quality and a second quality group indicating poor quality, step 103 can be as follows: Figure 3 As shown, it specifically includes: Step 301: Identify multiple target first answers corresponding to the first quality group and multiple target second answers corresponding to the second quality group from multiple answers; In this invention, multiple answers labeled with different quality groups are constructed for a single target question. To distinguish them by name, in this embodiment, the answer corresponding to the same group is called the target answer. Further, the target answer corresponding to the first quality group is called the first target answer, and the answer corresponding to the second quality group is called the second target answer. It is understood that the answer, target answer, first target answer, and second target answer are essentially the same; they are distinguished only by their names and do not imply any other limiting meaning.

[0036] Step 302: Calculate the first statistical score corresponding to the first answer of multiple targets and the second statistical score corresponding to the second answer of multiple targets respectively; In this embodiment of the invention, the first statistical score is used to characterize the overall quality of the first quality group, and the second statistical score is used to characterize the overall quality of the second quality group. By calculating the first statistical score and the second statistical score, the first statistical score and the second statistical score can be used as the measurement standard for the reward signal, thereby adjusting the first reward corresponding to each answer and guiding the reward model to output a more accurate reward.

[0037] It is understandable that after step 102, each answer receives a corresponding first reward. Therefore, multiple target first answers corresponding to the first quality group and multiple target second answers corresponding to the second quality group will also have corresponding target first rewards and target second rewards. In this embodiment of the invention, the first statistical score and the second statistical score are calculated based on the target first reward and the target second reward.

[0038] In an optional embodiment, the first statistical score is calculated in any of the following ways: taking the quantile of the target first reward corresponding to the multiple target first answers as the first statistical score; taking the average of the target first rewards corresponding to the multiple target first answers as the first statistical score; taking the ranking quality corresponding to the multiple target first answers as the first statistical score.

[0039] In a further optional embodiment, the second statistical score is calculated in any of the following ways: taking the quantile of the target second reward corresponding to the multiple target second answers as the second statistical score; taking the average of the target second rewards corresponding to the multiple target second answers as the second statistical score; taking the ranking quality corresponding to the multiple target second answers as the second statistical score.

[0040] Preferably, the first and second statistical scores are calculated using the same method. That is, when using the quantiles of the target first rewards corresponding to multiple target first answers as the first statistical score, the quantiles of the target second rewards corresponding to multiple target second answers are used as the second statistical score. This ensures that the first and second statistical scores represent the same level of overall quality, making them more meaningful for training. Specifically, the quantiles can be 25%, 50%, 75%, etc., preferably 50%, and the quantiles used for the target first reward and the target second reward should be the same.

[0041] Step 303: For each answer: Determine the reward signal corresponding to the answer based on the first reward, first statistical score, second statistical score, and preset reward rules.

[0042] This step can specifically include: for each target first answer: based on the relationship between the target first reward and the second statistical score corresponding to the target first answer, obtain the first reward signal corresponding to the target first answer; for each target second answer: based on the relationship between the target second reward and the first statistical score corresponding to the target second answer, obtain the second reward signal corresponding to the target second answer.

[0043] In other words, in this embodiment of the invention, for the target first answer corresponding to the first quality group, the reward direction is obtained by comparing it with the overall quality of the second quality group; similarly, for the target second answer corresponding to the second quality group, the reward direction is obtained by comparing it with the overall quality of the first quality group. It is understood that since the first quality group contains relatively high-quality answers, while the second quality group contains relatively low-quality answers, the overall quality of the first quality group should be higher than that of the second quality group. Therefore, through the calculation process of this embodiment, the reward model can be guided to output a higher reward for the target first answer in the first quality group, and a lower reward for the target second answer in the second quality group.

[0044] Specifically, based on the relationship between the target first reward and the second statistical score, a first reward signal corresponding to the target first answer is obtained, including: setting the first reward signal corresponding to the target first answer to a first threshold when the target first reward is greater than the second statistical score; and setting the first reward signal corresponding to the target first answer to a second threshold when the target first reward is less than or equal to the second statistical score. The first threshold can be set to 1, indicating that it conforms to human preferences and a reward is given; the second threshold can be set to -1, indicating that it does not conform to human preferences and a penalty is imposed. Similarly, based on the relationship between the target second reward and the first statistical score, a second reward signal corresponding to the target second answer is obtained, including: setting the second reward signal corresponding to the target second answer to a third threshold when the target second reward is less than the first statistical score; and setting the second reward signal corresponding to the target second answer to a fourth threshold when the target second reward is greater than or equal to the first statistical score. The third threshold can be set to 1, indicating that it conforms to human preferences and a reward is given; and the fourth threshold can be set to -1, indicating that it does not conform to human preferences and a penalty is imposed.

[0045] The following example illustrates steps 301 to 303: After constructing multiple training samples for the target answer, multiple answers {a1,a2,...,an, b1,b2,...,bn} are obtained. Each answer is labeled with a different quality group, namely the preferred group A = {a1,a2,...,an} and the rejected group B = {b1,b2,...,bn}. By inputting each answer and the target question into the original reward model, the first reward corresponding to each answer can be obtained. Then, the median of the first target rewards corresponding to all the first target answers a1,a2,...,an in the preferred group A = {a1,a2,...,an} is used to calculate the first statistical score T(S). c For each of the target second responses b1, b2, ..., bn in the rejection group B = {b1, b2, ..., bn}, the median of the target second reward is used to calculate the second statistical score T(S). r Furthermore, for each objective in the preferred group A, the reward r is calculated for the first answer ai. c If the AI's primary goal is reward s ai >T(S r (If the score of a single response is higher than the overall quality of Group B), then r c =1 (aligns with human preferences, reward), otherwise r c =-1 (not valid, penalty); calculate reward r for each second answer bj in the rejection group B. rIf bj's goal is the second reward s bj <T(S c (If a single answer score is lower than the overall quality of group A), then r r =1 (aligns with human preferences, reward), otherwise r r =-1 (not in compliance, penalty). This yields the reward signal corresponding to each first reward.

[0046] Step 104: Adjust the parameters of the original reward model according to the reward signal to obtain the reward model.

[0047] In this embodiment of the invention, the original reward model is the Bradley-Terry (BT) pairwise comparison model. After obtaining the reward signal corresponding to each reward, the parameters of the original reward model can be adjusted by calculating the preference probability of the BT model. The preference probability is calculated based on the reward signal corresponding to each first reward.

[0048] In summary, by labeling each answer with a different quality group and adjusting the first reward according to the second reward corresponding to the quality group, the embodiments of the present invention can avoid the computational bottleneck of the traditional pairwise generation reward model, while retaining the architecture of the single-point generation reward model, so that the trained reward model can output rewards based on a single answer.

[0049] In an optional embodiment, a format reward can be further added to the reward signal to ensure that the output format of the reward model conforms to human preferences. This is another training method for a reward model based on inter-group comparison provided in this embodiment of the invention. Figure 4 As shown, it includes: Step 401: Construct a training sample set; wherein, the training sample set includes a preset target question and multiple sets of training samples corresponding to the target question, and each set of training samples includes multiple answers labeled with different quality groups; Step 402: Input the target question and each answer into the original reward model respectively, and generate the reasoning text corresponding to each answer using a joint generation method; Step 403: Obtain the single-point first reward corresponding to each answer based on the reasoning text; Step 404: Identify multiple target first answers corresponding to the first quality group and multiple target second answers corresponding to the second quality group from the multiple answers; Step 405: Calculate the first statistical score corresponding to the first answer of multiple targets and the second statistical score corresponding to the second answer of multiple targets respectively; Step 406: For each target first answer: Based on the relationship between the target first reward and the second statistical score corresponding to the target first answer, obtain the first reward signal corresponding to the target first answer; For each target second answer: Based on the relationship between the target second reward and the first statistical score corresponding to the target second answer, obtain the second reward signal corresponding to the target second answer. Step 407: Based on whether the reasoning text meets the preset format rules, obtain the format reward corresponding to each answer; For example, if the preset format rules are met, the format reward r_f can be set to 0; if the preset format rules are not met, the format reward r_f can be set to -0.5.

[0050] Step 408: Calculate the total reward based on the first reward signal, the second reward signal, and the format reward; Specifically, the inter-group preference reward between the first quality group and the second quality group can be obtained based on the ratio of the first reward signal and the second reward signal; the sum of the inter-group preference reward and the format reward is used as the total reward.

[0051] Step 409: Adjust the parameters of the original reward model according to the total reward to obtain the reward model.

[0052] In summary, this embodiment of the invention groups multiple answers corresponding to the target question according to quality when constructing the training sample set to label the answers with quality groups. This allows the subsequent generation of reward signals based on the quality groups labeled with the answers and the preset reward rules to only compare at the group level, avoiding the computational bottleneck of traditional pairwise reward generation models. At the same time, it still uses the question-answer pair format of the target question and the answer as input, preserving the architecture of the single-point reward generation model. This allows the trained reward model to output rewards based on individual answers, and avoids the noise problem caused by subjective differences in traditional single-point reward generation models.

[0053] In one optional embodiment, the reward model trained by this invention can be combined with algorithms such as PPO / GRPO to jointly enhance the training of a large model. Specifically, the large model to be trained generates an answer, the trained reward model scores the answer, outputs a reward score, then PPO / GRPO is used to calculate the advantage function, and KL divergence multiplication is used to limit the update magnitude, so as to finally obtain the optimized large model.

[0054] The training apparatus for the reward model based on inter-group comparison provided by the present invention will be described below. The training apparatus for the reward model based on inter-group comparison described below and the training method for the reward model based on inter-group comparison described above can be referred to in correspondence.

[0055] like Figure 5 As shown, the training device 500 for the reward model based on inter-group comparison provided in this embodiment of the invention includes: The acquisition module 501 is used to construct a training sample set; wherein, the training sample set includes a preset target question and multiple sets of training samples corresponding to the target question, and each set of training samples includes multiple answers labeled with different quality groups; The generation module 502 is used to input the target question and each answer into the original reward model, and output the first reward corresponding to each answer; The determining module 503 is used to determine the reward signal corresponding to each of the first rewards based on the second rewards corresponding to multiple target answers that correspond to the same quality group in the answers and the preset reward rules. The adjustment module 504 is used to adjust the parameters of the original reward model according to the reward signal to obtain the reward model.

[0056] In an optional embodiment, the acquisition module 501 is further configured to generate prompt words based on a preset target question; the prompt words indicate the generation of multiple answers of different quality based on the target question; and repeatedly perform the following process to construct multiple sets of training samples corresponding to the target question: inputting the prompt words into a large model and outputting multiple answers corresponding to the target question; and labeling each answer with a quality group.

[0057] In an optional embodiment, the generation module 502 is further configured to input the target question and each of the answers into the original reward model respectively, and generate the reasoning text corresponding to each answer using a joint generation method; and obtain the single-point first reward corresponding to each answer based on the reasoning text.

[0058] In an optional embodiment, the quality grouping includes a first quality group indicating excellent quality and a second quality group indicating poor quality; the determining module 503 is further configured to: determine multiple target first answers corresponding to the first quality group and multiple target second answers corresponding to the second quality group from the multiple answers; calculate a first statistical score corresponding to the multiple target first answers and a second statistical score corresponding to the multiple target second answers respectively; and for each answer: determine a reward signal corresponding to the answer based on a first reward corresponding to the answer, the first statistical score, the second statistical score and a preset reward rule.

[0059] In an optional embodiment, the first statistical score is calculated in any of the following ways: taking the quantile of the target first reward corresponding to the plurality of target first answers as the first statistical score; taking the mean of the target first reward corresponding to the plurality of target first answers as the first statistical score; taking the ranking quality corresponding to the plurality of target first answers as the first statistical score.

[0060] In an optional embodiment, the second statistical score is calculated in any of the following ways: taking the quantile of the target second reward corresponding to the plurality of target second answers as the second statistical score; taking the mean of the target second reward corresponding to the plurality of target second answers as the second statistical score; taking the ranking quality corresponding to the plurality of target second answers as the second statistical score.

[0061] In an optional embodiment, the determining module 503 is further configured to, for each target first answer: obtain a first reward signal corresponding to the target first answer based on the size relationship between the target first reward corresponding to the target first answer and the second statistical score; For each target second answer: based on the relationship between the target second reward corresponding to the target second answer and the first statistical score, the second reward signal corresponding to the target second answer is obtained.

[0062] In an optional embodiment, the determining module 503 is further configured to, when the target first reward is greater than the second statistical score, set the first reward signal corresponding to the target first answer as a first threshold; and when the target first reward is less than or equal to the second statistical score, set the first reward signal corresponding to the target first answer as a second threshold.

[0063] In an optional embodiment, the determining module 503 is further configured to, when the target second reward is less than the first statistical score, set the second reward signal corresponding to the target second answer to a third threshold; and when the target second reward is greater than or equal to the first statistical score, set the second reward signal corresponding to the target second answer to a fourth threshold.

[0064] In an optional embodiment, the adjustment module 504 is further configured to: obtain the format reward corresponding to each of the answers based on whether the reasoning text satisfies the preset format rules; calculate the total reward based on the first reward signal, the second reward signal, and the format reward; and adjust the parameters of the original reward model based on the total reward to obtain the reward model.

[0065] In an optional embodiment, the adjustment module 504 is further configured to obtain the inter-group preference reward between the first quality group and the second quality group based on the ratio of the first reward signal and the second reward signal; and to use the sum of the inter-group preference reward and the format reward as the total reward.

[0066] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a training method for a reward model based on inter-group comparison. This method includes: constructing a training sample set; wherein the training sample set includes a preset target question and multiple sets of training samples corresponding to the target question, each set of training samples including multiple answers labeled with different quality groups; inputting the target question and each answer into an original reward model, outputting a first reward corresponding to each answer; determining a reward signal corresponding to each first reward based on the second rewards corresponding to multiple target answers corresponding to the same quality group and a preset reward rule; and adjusting the parameters of the original reward model according to the reward signal to obtain a reward model.

[0067] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0068] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the reward model based on inter-group comparison provided by the above methods. The method includes: constructing a training sample set; wherein the training sample set includes a preset target question and multiple sets of training samples corresponding to the target question, each set of training samples including multiple answers labeled with different quality groups; inputting the target question and each answer into an original reward model, and outputting a first reward corresponding to each answer; determining a reward signal corresponding to each first reward based on the second rewards corresponding to multiple target answers corresponding to the same quality group in the answer and a preset reward rule; and adjusting the parameters of the original reward model according to the reward signal to obtain a reward model.

[0069] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a training method for the reward model based on inter-group comparison provided by the methods described above. The method includes: constructing a training sample set; wherein the training sample set includes a preset target question and multiple sets of training samples corresponding to the target question, each set of training samples including multiple answers labeled with different quality groups; inputting the target question and each answer into an original reward model, and outputting a first reward corresponding to each answer; determining a reward signal corresponding to each first reward based on the second rewards corresponding to the multiple target answers corresponding to the same quality group and a preset reward rule; and adjusting the parameters of the original reward model according to the reward signal to obtain a reward model.

[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a reward model based on inter-group comparison, characterized in that, include: Construct a training sample set; wherein the training sample set includes a preset target question and multiple sets of training samples corresponding to the target question, and each set of training samples includes multiple answers labeled with different quality groups; The target question and each answer are input into the original reward model, and the first reward corresponding to each answer is output. Based on the second rewards corresponding to multiple target answers that correspond to the same quality group in the answers and the preset reward rules, determine the reward signal corresponding to each first reward; The original reward model is adjusted according to the reward signal to obtain the reward model.

2. The training method for the reward model based on inter-group comparison according to claim 1, characterized in that, The construction of training samples includes: Prompt words are generated based on a preset target question; the prompt words indicate the generation of multiple answers of different qualities based on the target question; Repeat the following process to construct multiple sets of training samples corresponding to the target question: input the prompt words into the large model and output multiple answers corresponding to the target question; label each answer with a quality group.

3. The training method for the reward model based on inter-group comparison according to claim 1, characterized in that, The step of inputting the target question and each answer into the original reward model, and outputting the first reward corresponding to each answer, includes: The target question and each of the answers are input into the original reward model respectively, and the reasoning text corresponding to each answer is generated by a joint generation method. Based on the reasoning text, a single-point first reward is obtained for each of the answers.

4. The training method for the reward model based on inter-group comparison according to claim 3, characterized in that, The quality grouping includes a first quality group indicating excellent quality and a second quality group indicating poor quality; the step of determining the reward signal corresponding to each first reward based on the second reward corresponding to multiple target answers in the same quality group and a preset reward rule includes: From the multiple answers, identify multiple target first answers corresponding to a first quality group and multiple target second answers corresponding to a second quality group; Calculate the first statistical score corresponding to the first answer for multiple targets and the second statistical score corresponding to the second answer for multiple targets respectively; For each answer: determine the reward signal corresponding to the answer based on the first reward, the first statistical score, the second statistical score, and the preset reward rules.

5. The training method for the reward model based on inter-group comparison according to claim 4, characterized in that, The first statistical score is calculated using any of the following methods: The quantile of the first reward corresponding to each of the multiple first answers to the target is taken as the first statistical score; The average of the target first rewards corresponding to the multiple target first answers is taken as the first statistical score; The ranking quality corresponding to the first answers of multiple objectives is taken as the first statistical score.

6. The training method for the reward model based on inter-group comparison according to claim 4, characterized in that, The second statistical score is calculated using any of the following methods: The quantile of the target second reward corresponding to each of the multiple target second answers is taken as the second statistical score; The average of the target second rewards corresponding to the multiple target second answers is taken as the second statistical score; The ranking quality corresponding to the second answers to the multiple objectives is taken as the second statistical score.

7. The training method for the reward model based on inter-group comparison according to claim 4, characterized in that, For each answer: determining the reward signal corresponding to the answer based on the first reward, the first statistical score, the second statistical score, and a preset reward rule, including: For each target first answer: based on the relationship between the target first reward corresponding to the target first answer and the second statistical score, the first reward signal corresponding to the target first answer is obtained; For each target second answer: based on the relationship between the target second reward corresponding to the target second answer and the first statistical score, the second reward signal corresponding to the target second answer is obtained.

8. The training method for the reward model based on inter-group comparison according to claim 7, characterized in that, The step of obtaining the first reward signal corresponding to the first target answer based on the relationship between the first target reward and the second statistical score includes: If the target first reward is greater than the second statistical score, the first reward signal corresponding to the target first answer is set as a first threshold. If the target first reward is less than or equal to the second statistical score, the first reward signal corresponding to the target first answer is set as the second threshold. And / or, The step of obtaining the second reward signal corresponding to the second target answer based on the relationship between the second target reward and the first statistical score includes: If the target second reward is less than the first statistical score, the second reward signal corresponding to the target second answer is set as the third threshold. If the target second reward is greater than or equal to the first statistical score, the second reward signal corresponding to the target second answer is set as the fourth threshold.

9. The training method for the reward model based on inter-group comparison according to claim 7, characterized in that, Also includes: Based on whether the reasoning text satisfies the preset format rules, the format reward corresponding to each of the answers is obtained; The step of adjusting the parameters of the original reward model based on the reward signal to obtain the reward model includes: The total reward is calculated based on the first reward signal, the second reward signal, and the formatted reward. The original reward model is adjusted based on the total reward to obtain the reward model.

10. A training device for a reward model based on inter-group comparison, characterized in that, include: An acquisition module is used to construct a training sample set; wherein, the training sample set includes a preset target question and multiple sets of training samples corresponding to the target question, and each set of training samples includes multiple answers labeled with different quality groups; The generation module is used to input the target question and each answer into the original reward model, and output the first reward corresponding to each answer; The determination module is used to determine the reward signal corresponding to each of the first rewards based on the second rewards corresponding to multiple target answers that correspond to the same quality group in the answers and the preset reward rules. The adjustment module is used to adjust the parameters of the original reward model according to the reward signal to obtain the reward model.

Images