A reliable reward evaluation method and device that combines human preferences with verifiable correctness signals

Abstract

This application provides a reliable reward evaluation method and apparatus that combines human preferences with verifiable correctness signals, relating to the field of artificial intelligence technology. The method includes: inputting the response of the model to be evaluated into a basic reward model and a verifiable correctness signal reward model, respectively, to obtain a reward score output by each model; performing a weighted sum based on the reward scores output by each model and the weights of each model to obtain a final reward score for the response of the model to be evaluated; the basic reward model is used to represent human preference scores for the model response; the verifiable correctness signal reward model is used to represent the correctness of the model response in a specific aspect. The reliable reward evaluation method combining human preferences and verifiable correctness signals provided in this application not only considers human preferences but also comprehensively considers verifiable correctness signals such as factuality and instruction compliance in the model response, thereby providing more accurate and reliable rewards.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a reliable reward evaluation method and apparatus that combines human preferences with verifiable correctness signals. Background Technology

[0002] In the field of modern artificial intelligence, large-scale language models have become core tools for natural language processing and generation tasks. These models learn from large amounts of text data and can generate high-quality model responses, which are widely used in various scenarios such as chatbots, text generation, and machine translation.

[0003] To further improve the performance of large language models, reward models are widely used in their training. In these techniques, reward models are primarily trained based on human preference data. These models learn how to evaluate and score the quality of model responses by collecting data on human preferences for different responses. This preference data is used to train reward models, enabling them to automatically evaluate the quality of model responses and providing guidance for the training of large language models.

[0004] However, due to the subjectivity of human preferences, different people may have completely different evaluations of the same response. Therefore, reward scores based on human preference data may be biased by subjectivity when reward models evaluate model responses, tending to select responses with better language style and longer length, while ignoring the actual content and accuracy of the model response. Summary of the Invention

[0005] The purpose of this application is to provide a reliable reward evaluation method and apparatus that combines human preferences with verifiable correctness signals. This method not only considers human preferences but also comprehensively considers verifiable correctness signals such as factual and instruction-following behavior of the model response, thereby providing more accurate and reliable rewards.

[0006] This application provides a reliable reward evaluation method that combines human preferences with verifiable correctness signals, including:

[0007] The response of the model to be evaluated is obtained and input into the basic reward model and the verifiable correctness signal reward model respectively to obtain the reward score output by each model; the reward score output by each model and the weight of each model are weighted and summed to obtain the final reward score for the response of the model to be evaluated; wherein, the basic reward model is used to represent the human preference score for the model response; the verifiable correctness signal reward model is used to represent the correctness of the model response in a specific aspect.

[0008] Optionally, the step of inputting the response of the model to be evaluated into the basic reward model and the verifiable correctness signal reward model respectively to obtain the reward score output by each model includes: inputting the response of the model to be evaluated into the basic reward model to obtain the human preference score for the response of the model to be evaluated, and determining the preference score as the reward score output by the basic reward model; wherein, the basic reward model is a regression model.

[0009] Optionally, the verifiable correctness signal reward model includes: a router and multiple verification agents; the verification agents include: a factual verification agent and an instruction compliance verification agent; the factual verification agent is used to evaluate whether the factual information contained in the model response is correct; the instruction compliance verification agent is used to evaluate whether the model response meets the hard constraints specified in the instruction; the step of inputting the model response to be evaluated into the basic reward model and the verifiable correctness signal reward model respectively to obtain the reward score output by each model includes: generating a corresponding verification instruction based on the model response to be evaluated, and inputting the verification instruction into the router; selecting a verification agent through the router, and filtering out the target agent to be used from the multiple verification agents; the target agent includes at least one verification agent; using the target agent to verify the correctness of the model response to be evaluated, and obtaining the reward score output by the target agent.

[0010] Optionally, the step of inputting the verification command into the router and selecting the verification proxy through the router to filter out the target proxy to be used from the plurality of verification proxies includes: inputting the verification command and the description information of each of the plurality of verification proxies into a natural language model, and having the natural language model filter out the matching target proxy from the plurality of verification proxies according to the input command and the description information of each verification proxy; wherein, the description information includes: the function and usage conditions of the verification proxy.

[0011] Optionally, the target agent includes: the fact verification agent; the model response to be evaluated includes: two different model responses; the step of using the target agent to verify the correctness of the model response to be evaluated and obtaining the reward score output by the target agent includes: inputting the two different model responses into the fact verification agent, identifying the differences between the two different model responses in the claimed facts, obtaining a difference identification result, and generating a corresponding difference query based on the difference identification result; using the difference query to perform evidence retrieval, and performing fact verification on the differences between the two different model responses based on the retrieved evidence, and scoring each of the two different model responses according to the verification result to obtain a score for each model response; determining the reward score of the fact verification agent for the model response to be evaluated based on the score of each model response; wherein, the score of the model response with correct facts is higher than the score of the model response with incorrect facts.

[0012] Optionally, the target agent includes: the instruction compliance verification agent; the step of using the target agent to verify the correctness of the response of the model to be evaluated and obtaining a reward score output by the target agent includes: extracting at least one constraint condition for the model response from the verification instruction, and generating a corresponding constraint verification script based on the constraint condition; using the constraint verification script to perform constraint verification on the response of the model to be evaluated, determining whether the response of the model to be evaluated satisfies each of the at least one constraint condition, and obtaining a constraint verification result; calculating the average score of each constraint condition in the constraint verification result to obtain the reward score of the instruction compliance verification agent for the response of the model to be evaluated.

[0013] Optionally, the step of using the target agent to verify the correctness of the response of the model to be evaluated and obtaining the reward score output by the target agent includes: weighting and summing the reward scores output by each of the plurality of verification agents to obtain the reward score output by the target agent; wherein, the reward score output by the target agent is: the reward score output by the verifiable correctness signal reward model for the response of the model to be evaluated.

[0014] This application also provides a reliable reward evaluation device that combines human preferences with verifiable correctness signals, including:

[0015] The evaluation module is used to acquire the response of the model to be evaluated; the reward scoring module is used to input the response of the model to be evaluated into the basic reward model and the verifiable correctness signal reward model respectively to obtain the reward score output by each model; the reward scoring module is also used to perform a weighted sum based on the reward score output by each model and the weight of each model to obtain the final reward score for the response of the model to be evaluated; wherein, the basic reward model is used to represent the human preference rating for the model response; the verifiable correctness signal reward model is used to represent the correctness of the model response in a specific aspect.

[0016] Optionally, the reward scoring module is specifically used to input the response of the model to be evaluated into the basic reward model to obtain the human preference score for the response of the model to be evaluated, and to determine the preference score as the reward score output by the basic reward model; wherein, the basic reward model is a regression model.

[0017] Optionally, the verifiable correctness signal reward model includes: a router and multiple verification agents; the verification agents include: a factual verification agent and an instruction compliance verification agent; the factual verification agent is used to evaluate whether the factual information contained in the model response is correct; the instruction compliance verification agent is used to evaluate whether the model response meets the hard constraints specified in the instruction; the reward scoring module is specifically used to generate corresponding verification instructions based on the model response to be evaluated, and input the verification instructions into the router, and select the verification agent through the router, filtering out the target agent to be used from the multiple verification agents; the target agent includes at least one verification agent; the reward scoring module is further specifically used to use the target agent to perform correctness verification on the model response to be evaluated, and obtain the reward score output by the target agent.

[0018] Optionally, the reward scoring module is specifically used to input the verification instruction and the description information of each verification agent among the plurality of verification agents into a natural language model, and the natural language model selects a matching target agent from the plurality of verification agents based on the input instruction and the description information of each verification agent; wherein, the description information includes: the function and usage conditions of the verification agent.

[0019] Optionally, the target agent includes: the fact verification agent; the model response to be evaluated includes: two different model responses; the reward scoring module is specifically used to input the two different model responses into the fact verification agent, identify the differences between the two different model responses in the claimed facts, obtain a difference identification result, and generate a corresponding difference query based on the difference identification result; the reward scoring module is further used to use the difference query to perform evidence retrieval, and to perform fact verification on the differences between the two different model responses based on the retrieved evidence, and to score each of the two different model responses according to the verification result, to obtain a score for each model response; the reward scoring module is further used to determine the reward score of the fact verification agent for the model response to be evaluated based on the score of each model response; wherein, the score of the model response that is factually correct is higher than the score of the model response that is factually incorrect.

[0020] Optionally, the target agent includes: the instruction compliance verification agent; the reward scoring module, specifically configured to extract at least one constraint condition for the model response from the verification instruction, and generate a corresponding constraint verification script based on the constraint condition; the reward scoring module is further configured to use the constraint verification script to perform constraint verification on the model response to be evaluated, determine whether the model response to be evaluated satisfies each of the at least one constraint condition, and obtain a constraint verification result; the reward scoring module is further configured to calculate the average score of each constraint condition in the constraint verification result, and obtain a reward score from the instruction compliance verification agent for the model response to be evaluated.

[0021] Optionally, the reward scoring module is specifically used to obtain the reward score output by the target agent by weighted summation of the reward scores output by each of the plurality of verification agents; wherein, the reward score output by the target agent is: the reward score output by the verifiable correctness signal reward model for the response of the model to be evaluated.

[0022] This application also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of a reliable reward evaluation method combining human preferences and verifiable correctness signals as described above.

[0023] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the reliable reward evaluation method combining human preferences and verifiable correctness signals as described above.

[0024] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the reliable reward evaluation method combining human preferences and verifiable correctness signals as described above.

[0025] The reliable reward evaluation method and apparatus provided in this application, which combines human preferences and verifiable correctness signals, firstly acquires the response of the model to be evaluated and inputs it into a basic reward model and a verifiable correctness signal reward model, respectively, to obtain a reward score output by each model. Then, a weighted sum is performed based on the reward scores output by each model and the weights of each model to obtain a final reward score for the response of the model to be evaluated. The basic reward model represents human preference scores for the model response, and the verifiable correctness signal reward model represents the correctness of the model response in a specific aspect. In this way, not only human preferences are considered, but also verifiable correctness signals such as factuality and instruction compliance of the model response are comprehensively considered, thereby providing more accurate and reliable rewards. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the reward system provided in the embodiments of this application;

[0028] Figure 2 This is a flowchart illustrating the reliable reward evaluation method that combines human preferences with verifiable correctness signals provided in this application.

[0029] Figure 3 This is a schematic diagram of the benchmark test results of the reward model provided in this application;

[0030] Figure 4 This is a schematic diagram of the best candidate search results for the reward model provided in this application;

[0031] Figure 5 This is a schematic diagram illustrating the results of using REWARDAGENT to construct training preference pairs and train a large language model, as provided in this application.

[0032] Figure 6 This is a schematic diagram of the structure of the reliable reward evaluation device that combines human preferences with verifiable correctness signals provided in this application;

[0033] Figure 7This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0036] While reward models based on human preferences have achieved some success in training large language models, they also have some significant drawbacks. First, human preferences are inherently subjective; different people may rate the same response differently. This subjectivity can lead to biases in reward models when evaluating responses, favoring those with better language style and greater length, while ignoring the actual content and accuracy of the response. Second, existing reward models primarily focus on human preferences, neglecting crucial information such as factual accuracy and adherence to instructions. For example, in a task requiring the generation of text about a historical event, a response might be fluent and lengthy, but contain inaccurate historical facts. Existing reward models might award a higher score based on language style, ignoring these factual errors. Similarly, in tasks requiring strict adherence to instructions, a response might meet most requirements but miss some key instructional constraints. Existing reward models may fail to accurately identify these instructional adherence issues, thus affecting the evaluation of the response. This neglect of factual information and instructional compliance not only affects the reliability of reward models, but may also lead to erroneous or unacceptable responses in large language models trained in real-world applications, impacting user experience and model credibility.

[0037] To address the aforementioned technical problems in related technologies, this application provides a reliable reward evaluation method that combines human preferences with verifiable correctness signals. This method is applied to, for example... Figure 1 The reward evaluation system shown is as follows: Figure 1 As shown, the system comprises a basic reward model and a verifiable correctness signal reward model (containing a router and multiple verification agents). After inputting the model response into both models, a reward score is obtained from the output of each model, and the final reward score is obtained through weighted summation. This method not only considers human preferences but also comprehensively considers verifiable correctness signals such as factuality and instruction compliance in the response, thus providing more accurate and reliable rewards. Through this comprehensive evaluation method, this invention can effectively improve the training effect and performance of large-scale language models, enabling them to generate higher-quality and more compliant text responses in various natural language processing tasks.

[0038] The following description, in conjunction with the accompanying drawings, details the reliable reward evaluation method that combines human preferences and verifiable correctness signals provided in this application through specific embodiments and application scenarios.

[0039] like Figure 2 As shown in the embodiment of this application, a reliable reward evaluation method combining human preferences and verifiable correctness signals is provided. This method may include the following steps 201 and 202:

[0040] Step 201: Obtain the response of the model to be evaluated, and input the response of the model to be evaluated into the basic reward model and the verifiable correctness signal reward model respectively to obtain the reward score output by each model.

[0041] The basic reward model is used to characterize human preference ratings for model responses; the verifiable correctness signal reward model is used to characterize the correctness of model responses in specific aspects.

[0042] For example, the above-mentioned model response to be evaluated is the text response output by the large language model. For this model response to be evaluated, it needs to be scored by a reward model so that the large language model can be trained based on the scoring results, thereby helping the large language model learn to generate text responses that are more in line with human expectations.

[0043] For example, in this embodiment of the application, a traditional reward model based on human preferences is combined with verifiable correctness signals from different sources. The core idea is shown in the following formula:

[0044] r(x,y)=λ⋅rRM(x,y)+∑i∈Axwi⋅ai(x,y) (Formula 1)

[0045] Where r(x,y) is the final reward score, representing the comprehensive evaluation of instruction x and response y. λ: The weight of the base reward model, used to adjust the contribution of human preference rewards to the final reward. rRM(x,y): The output of the base reward model, representing the human preference score for response y. Ax: The set of indices of verification agents associated with instruction x, representing the verification agents that need to be invoked based on instruction x. wi: The weight of the i-th verification agent, used to adjust the contribution of each verification agent to the final reward. ai(x,y): The output of the i-th verification agent, representing the correctness score of response y in a specific aspect.

[0046] For example, according to Formula 1 above, the reward score in this embodiment may include the following two parts:

[0047] ① Basic reward model part: λ⋅rRM(x,y), this part represents the contribution of human preference reward. rRM(x,y) is the output of a regression model, usually trained based on human-labeled preference data. It reflects the subjective evaluation of human response y. The weight λ is used to adjust the importance of this part in the final reward.

[0048] ② Verifiable Correctness Signal Part: ∑i∈Axwi⋅ai(x,y), this part represents the contribution of the verifiable correctness signal. Ax is a set of verification agents dynamically selected according to the instruction x. Each verification agent ai evaluates the correctness of the response y in a specific aspect. For example, ai can be a factual verification agent or an instruction compliance verification agent. The output ai(x,y) of each verification agent is multiplied by its corresponding weight wi, and then the contributions of all verification agents are summed to obtain this part.

[0049] It should be noted that, in specific implementation, this application embodiment designs a reward agent called REWARDAGENT, which integrates human preference rewards and two verifiable signals: factual and instruction compliance. REWARDAGENT consists of three main modules: Router, Verification Agents, and Judger.

[0050] Specifically, step 201 above, the reward score calculation step for the basic reward model part, may include the following step 201a:

[0051] Step 201a: Input the response of the model to be evaluated into the basic reward model to obtain the human preference score for the response of the model to be evaluated, and determine the preference score as the reward score output by the basic reward model.

[0052] The basic reward model is a regression model.

[0053] For example, in this embodiment, human preferences are still included as part of the reward scoring. The base reward model learns how to evaluate and score the quality of the model's responses by collecting human preference choices for different responses. For instance, in a text generation task, human annotators might compare multiple generated text responses and select the response they deem to be of higher quality. This preference data is used to train the reward model, enabling it to automatically evaluate the quality of responses and providing guidance for the training of large language models.

[0054] For example, regarding the reward score for the verifiable correctness signal portion mentioned above, the verifiable correctness signal reward model includes: a router and multiple verification agents; the verification agents include: a factual verification agent and an instruction compliance verification agent; the factual verification agent is used to evaluate whether the factual information contained in the model response is correct; the instruction compliance verification agent is used to evaluate whether the model response meets the hard constraints specified in the instruction.

[0055] Specifically, in step 201 above, the reward score calculation step for the verifiable correctness signal reward model part may include the following steps 201b and 201c:

[0056] Step 201b: Generate a corresponding verification instruction based on the response of the model to be evaluated, and input the verification instruction into the router. The router selects a verification agent and filters out the target agent to be used from the multiple verification agents. The target agent includes at least one verification agent.

[0057] Step 201c: Use the target agent to verify the correctness of the response of the model to be evaluated, and obtain the reward score output by the target agent.

[0058] For example, in this embodiment of the application, the router is the first module of the REWARDAGENT, and its main function is to analyze the input instructions and determine which authentication agents need to be invoked to evaluate the response. Different instructions may require evaluation of different aspects of the response, so dynamically selecting the appropriate authentication agent can reduce inference costs and reduce potential accumulated errors.

[0059] Specifically, step 201b above, the step of selecting an authentication agent for the router, may further include the following step 201b1:

[0060] Step 201b1: Input the verification instruction and the description information of each verification agent in the plurality of verification agents into the natural language model. The natural language model then selects the matching target agent from the plurality of verification agents based on the input instruction and the description information of each verification agent.

[0061] The description information includes: the functionality and usage conditions of the verification agent.

[0062] For example, the router uses an existing large language model as its core. By inputting instructions and descriptions of all verification proxies, the large language model selects the appropriate verification proxy. Specifically, we first provide a concise description for each verification proxy, outlining its function and usage conditions. Then, the instructions and all proxy descriptions are input into the large language model, which selects the appropriate verification proxy based on the requirements of the instructions. For instance, if the instruction requires generating text about a historical event and ensuring the accuracy of facts within the text, the router will select a factual verification proxy; if the instruction includes specific format or length requirements, the router will select an instruction-compliant verification proxy.

[0063] For example, the verification proxy is a core component of the REWARDAGENT, used to evaluate the correctness of the response in various aspects. In this embodiment, we designed two main verification proxies: a factual verification proxy and a command compliance verification proxy. It should be noted that, in addition to the two verification proxies mentioned above, other proxies can be added to the model according to the actual situation. The router can select one or more verification proxies based on the input commands, or it can choose not to select any verification proxies.

[0064] In one possible implementation, the target agent includes: the factual verification agent; the model response to be evaluated includes: two different model responses.

[0065] Specifically, when the target agent includes a fact-verification agent, step 201c may further include steps 201c1 to 201c3:

[0066] Step 201c1: Input the two different model responses into the factual verification agent, identify the differences between the two different model responses in the claimed facts, obtain the difference identification result, and generate the corresponding difference query based on the difference identification result.

[0067] Step 201c2: Use the difference query to retrieve evidence, and perform fact verification on the difference between the two different model responses based on the retrieved evidence. Then, score each of the two different model responses according to the verification results to obtain a score for each model response.

[0068] Step 201c3: Based on the score of each model response, determine the reward score of the factual verification agent for the response of the model to be evaluated.

[0069] Among them, the model response that is factually correct scores higher than the model response that is factually incorrect.

[0070] For example, the primary task of a factual verification agent is to evaluate whether the factual information contained in a response is correct. To accomplish this task efficiently, we designed a factual verification agent based on pairwise comparisons. This agent evaluates the factuality of a response through the following four main steps:

[0071] Difference Proposal: Identifies key differences in the claimed facts between two given responses. For example, if one response claims the height of Mount Everest is 8848 meters, while another response claims it is 8868 meters, the difference proposal step will identify the difference between these two height values.

[0072] Query Generation: This step generates queries based on the identified differences to retrieve evidence that helps distinguish between them. For example, given the height differences mentioned above, the query generation step might produce a query such as "What is the height of Mount Everest?".

[0073] Evidence Generation: Using generated queries, evidence is supported by parametric knowledge retrieval from external search engines or large language models. For example, a search engine might retrieve the height of Mount Everest as 8848 meters, or this fact might be verified using the knowledge base within a large language model.

[0074] Verification: Based on the collected evidence and the original response, assign an integer score from 0 to 1 to each response. A higher score is given if the facts in the response are verified to be correct; a lower score is given if the facts are incorrect.

[0075] This pairwise comparison-based factual verification agent can effectively capture subtle factual differences between responses while significantly reducing inference time costs because it only needs to verify the differences between responses, rather than all claimed facts.

[0076] In another possible implementation, the target agent includes: the instruction follows a verification agent.

[0077] Specifically, when the target agent includes instructions that comply with the verification agent, step 201c may further include steps 201c4 to 201c6:

[0078] Step 201c4: Extract at least one constraint condition for the model response from the verification instruction, and generate a corresponding constraint verification script based on the constraint condition.

[0079] Step 201c5: Use the constraint verification script to perform constraint verification on the response of the model to be evaluated, determine whether the response of the model to be evaluated satisfies each of the at least one constraint condition, and obtain the constraint verification result.

[0080] Step 201c6: Calculate the average score of each constraint condition in the constraint verification results to obtain the reward score of the instruction compliance verification agent for the response of the model to be evaluated.

[0081] For example, the primary task of an Instruction-Following Verification Agent is to evaluate whether the response meets the hard constraints specified in the instruction. These constraints typically include length limits, format requirements, keyword restrictions, etc. The Instruction-Following Verification Agent accomplishes its task through the following three steps:

[0082] Constraint parsing: Extracts hard constraints from instructions. For example, if the instruction requires generating text of no more than 100 characters, the constraint parsing step will extract the constraint "no more than 100 characters".

[0083] Code Generation and Refinement: Generates a Python script to check if the response meets the extracted constraints. The generated code takes the response as input and returns a boolean value indicating whether the response satisfies the constraints. For example, for the length constraint mentioned above, the generated code might check if the number of words in the response does not exceed 100. If an error occurs during code execution, the error message and the original code are fed back to the model to generate more accurate code.

[0084] Verification: Execute the generated Python code to obtain a binary score (0 or 1) indicating whether the response satisfies each hard constraint. The final score is the average of all hard constraint scores.

[0085] This instruction-based verification agent can effectively evaluate whether the response meets the hard constraints in the instruction, thereby ensuring that the generated text meets the user's requirements.

[0086] Step 202: Perform a weighted summation based on the reward score output by each model and the weight of each model to obtain the final reward score for the response of the model to be evaluated.

[0087] For example, after obtaining the reward score for each model output, the final reward score can be calculated using a judge. The judge is the last module of the REWARDAGENT, and its main function is to integrate the correctness signal from the verification agent and the human preference score from the reward model using Formula 1 above to generate the final reward score. In this implementation, we use a weighted sum as the judge, where λ and wi represent the weights of the base reward model and each verification agent, respectively. These weights can be adjusted according to specific circumstances to adapt to different application scenarios. For example, in some scenarios, factuality may be more important, so the weight of the factual verification agent can be increased; in other scenarios, human preference may be more critical, so the weight of the reward model can be increased. In this way, the judge can comprehensively consider human preference and verifiable correctness signals to provide a comprehensive and accurate reward score for each response.

[0088] The reliable reward evaluation method combining human preferences and verifiable correctness signals provided in this application first obtains the response of the model to be evaluated and inputs it into a basic reward model and a verifiable correctness signal reward model respectively, obtaining a reward score output by each model. Then, a weighted sum is performed based on the reward scores output by each model and the weights of each model to obtain a final reward score for the response of the model to be evaluated. The basic reward model represents human preference scores for the model response, and the verifiable correctness signal reward model represents the correctness of the model response in a specific aspect. In this way, not only human preferences are considered, but also verifiable correctness signals such as factuality and instruction compliance of the model response are comprehensively considered, thereby providing more accurate and reliable rewards.

[0089] By testing on existing reward model benchmarks (such as...) Figure 3 (as shown) and the best candidate search for inference time in real-world downstream tasks (such as...) Figure 4As shown in the example, REWARDAGENT significantly outperforms traditional reward models. For instance, as... Figure 3 As shown, in benchmarks such as RM-Bench and JudgeBench, REWARDAGENT's accuracy significantly outperforms existing reward models, such as ArmoRM and INF-ORM-Llama3.1-70B. Furthermore, in IFBench, a newly constructed benchmark for evaluating instruction compliance, REWARDAGENT also performs exceptionally well, especially in subsets with numerous hard constraints, where its accuracy far surpasses other models. These results demonstrate that REWARDAGENT can more accurately evaluate response quality and select responses that better meet requirements. Moreover, as... Figure 5 As shown, using REWARDAGENT to construct training preference pairs and train large language models can also achieve better performance than traditional reward models in various natural language processing benchmarks. For example, ... Figure 5 As shown, in tasks such as MMLU, MMLU-Pro, TriviaQA, and TruthfulQA, large language models trained using REWARDAGENT outperform those trained using traditional reward models in both accuracy and reliability. This further demonstrates the effectiveness of this invention in improving the training effect and performance of large language models. By comprehensively considering human preferences and verifiable correctness signals, this invention can effectively improve the performance of reward models, thereby enhancing the training effect and application performance of large language models, possessing significant practical application value and broad application prospects.

[0090] The key point of the reliable reward evaluation method combining human preferences and verifiable correctness signals provided in this application is the proposal of a reward system architecture that integrates human preferences with verifiable correctness signals. This architecture, by introducing verifiable correctness signals such as factual information and instruction compliance, compensates for the shortcomings of existing reward models in terms of factual information and instruction compliance, thereby providing more accurate and reliable rewards. Furthermore, this invention designs a flexible reward agent framework, including a router, a verification agent, and a judge, enabling the reward system to dynamically select an appropriate verification agent based on different instructions and comprehensively consider multiple signals to evaluate the quality of the response. This comprehensive evaluation method not only improves the reliability and accuracy of the reward model but also enhances its applicability and robustness in various natural language processing tasks.

[0091] It should be noted that the reliable reward evaluation method combining human preferences and verifiable correctness signals provided in this application embodiment can be executed by a reliable reward evaluation device combining human preferences and verifiable correctness signals, or by a control module within that device for executing the method. This application embodiment uses the execution of the reliable reward evaluation method combining human preferences and verifiable correctness signals by a reliable reward evaluation device as an example to illustrate the reliable reward evaluation device combining human preferences and verifiable correctness signals provided in this application embodiment.

[0092] It should be noted that, in the embodiments of this application, the reliable reward evaluation methods combining human preferences and verifiable correctness signals shown in the accompanying drawings are all illustrated using one accompanying drawing from one of the embodiments of this application as an example. In specific implementations, the reliable reward evaluation methods combining human preferences and verifiable correctness signals shown in the accompanying drawings of the above methods can also be implemented in conjunction with any other accompanying drawings illustrated in the above embodiments, which will not be elaborated here.

[0093] The reliable reward evaluation device that combines human preferences and verifiable correctness signals provided in this application is described below. The reliable reward evaluation method that combines human preferences and verifiable correctness signals described above can be referred to in correspondence with each other.

[0094] Figure 6 A schematic diagram of the structure of the reliable reward evaluation device that combines human preferences and verifiable correctness signals provided in the embodiments of this application is shown below. Figure 6 As shown, it specifically includes:

[0095] The acquisition module 601 is used to acquire the response of the model to be evaluated; the reward scoring module 602 is used to input the response of the model to be evaluated into the basic reward model and the verifiable correctness signal reward model respectively to obtain the reward score output by each model; the reward scoring module 602 is also used to perform a weighted sum based on the reward score output by each model and the weight of each model to obtain the final reward score for the response of the model to be evaluated; wherein, the basic reward model is used to represent the human preference score for the model response; the verifiable correctness signal reward model is used to represent the correctness of the model response in a specific aspect.

[0096] Optionally, the reward scoring module 602 is specifically used to input the response of the model to be evaluated into the basic reward model to obtain the human preference score for the response of the model to be evaluated, and to determine the preference score as the reward score output by the basic reward model; wherein, the basic reward model is a regression model.

[0097] Optionally, the verifiable correctness signal reward model includes: a router and multiple verification agents; the verification agents include: a factual verification agent and an instruction compliance verification agent; the factual verification agent is used to evaluate whether the factual information contained in the model response is correct; the instruction compliance verification agent is used to evaluate whether the model response meets the hard constraints specified in the instruction; the reward scoring module 602 is specifically used to generate corresponding verification instructions based on the model response to be evaluated, and input the verification instructions into the router, and select the verification agent through the router, filtering out the target agent to be used from the multiple verification agents; the target agent includes at least one verification agent; the reward scoring module 602 is further used to use the target agent to perform correctness verification on the model response to be evaluated, and obtain the reward score output by the target agent.

[0098] Optionally, the reward scoring module 602 is specifically used to input the verification instruction and the description information of each verification agent among the plurality of verification agents into a natural language model, and the natural language model selects a matching target agent from the plurality of verification agents according to the input instruction and the description information of each verification agent; wherein, the description information includes: the function and usage conditions of the verification agent.

[0099] Optionally, the target agent includes: the fact verification agent; the model response to be evaluated includes: two different model responses; the reward scoring module 602 is specifically used to input the two different model responses into the fact verification agent, identify the differences between the two different model responses in the claimed facts, obtain a difference identification result, and generate a corresponding difference query based on the difference identification result; the reward scoring module 602 is further used to use the difference query to perform evidence retrieval, and to perform fact verification on the differences between the two different model responses based on the retrieved evidence, and to score each of the two different model responses according to the verification result, to obtain a score for each model response; the reward scoring module 602 is further used to determine the reward score of the fact verification agent for the model response to be evaluated based on the score of each model response; wherein, the score of the model response that is factually correct is higher than the score of the model response that is factually incorrect.

[0100] Optionally, the target agent includes: the instruction compliance verification agent; the reward scoring module 602, specifically configured to extract at least one constraint condition for the model response from the verification instruction, and generate a corresponding constraint verification script based on the constraint condition; the reward scoring module 602 is further configured to use the constraint verification script to perform constraint verification on the model response to be evaluated, determine whether the model response to be evaluated satisfies each of the at least one constraint condition, and obtain a constraint verification result; the reward scoring module 602 is further configured to calculate the average score of each constraint condition in the constraint verification result, and obtain the reward score of the instruction compliance verification agent for the model response to be evaluated.

[0101] Optionally, the reward scoring module 602 is specifically used to perform a weighted summation of the reward scores output by each of the plurality of verification agents to obtain the reward score output by the target agent; wherein, the reward score output by the target agent is: the reward score output by the verifiable correctness signal reward model for the response of the model to be evaluated.

[0102] The reliable reward evaluation device combining human preferences and verifiable correctness signals provided in this application first acquires the response of the model to be evaluated and inputs it into a basic reward model and a verifiable correctness signal reward model respectively, obtaining a reward score output by each model. Then, a weighted sum is performed based on the reward scores output by each model and the weight of each model to obtain a final reward score for the response of the model to be evaluated. The basic reward model represents human preference scores for the model response, and the verifiable correctness signal reward model represents the correctness of the model response in a specific aspect. In this way, not only human preferences are considered, but also verifiable correctness signals such as factuality and instruction compliance of the model response are comprehensively considered, thereby providing more accurate and reliable rewards.

[0103] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a reliable reward evaluation method that combines human preferences and verifiable correctness signals. This method includes: first, acquiring the response of the model to be evaluated and inputting the response into a basic reward model and a verifiable correctness signal reward model, respectively, to obtain a reward score output by each model; then, performing a weighted sum based on the reward scores output by each model and the weights of each model to obtain a final reward score for the response of the model to be evaluated; wherein the basic reward model is used to characterize human preference ratings for the model response; and the verifiable correctness signal reward model is used to characterize the correctness of the model response in a specific aspect. In this way, not only can human preferences be taken into account, but verifiable correctness signals such as factual evidence and instruction compliance in the model's response can also be considered, thus providing more accurate and reliable rewards.

[0104] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0105] On the other hand, this application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the reliable reward evaluation method provided by the above methods, which combines human preferences and verifiable correctness signals. This method includes: first, acquiring the response of a model to be evaluated, and inputting the response into a basic reward model and a verifiable correctness signal reward model respectively, to obtain a reward score output by each model; then, performing a weighted sum based on the reward scores output by each model and the weights of each model to obtain a final reward score for the response of the model to be evaluated; wherein, the basic reward model is used to characterize human preference scores for the model response; and the verifiable correctness signal reward model is used to characterize the correctness of the model response in a specific aspect. Thus, not only human preferences can be considered, but also verifiable correctness signals such as factuality and instruction compliance of the model response can be comprehensively considered, thereby providing more accurate and reliable rewards.

[0106] Furthermore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned reliable reward evaluation methods combining human preferences and verifiable correctness signals. This method includes: first, acquiring the response of a model to be evaluated and inputting the response into a basic reward model and a verifiable correctness signal reward model, respectively, to obtain a reward score output by each model; then, performing a weighted sum based on the reward scores output by each model and the weights of each model to obtain a final reward score for the response of the model to be evaluated; wherein the basic reward model is used to characterize human preference scores for the model response; and the verifiable correctness signal reward model is used to characterize the correctness of the model response in a specific aspect. Thus, not only human preferences can be considered, but verifiable correctness signals such as factuality and instruction compliance in the model response can also be comprehensively considered, thereby providing more accurate and reliable rewards.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A reliable reward evaluation method that combines human preferences with verifiable correctness signals, characterized in that, include: Obtain the response of the model to be evaluated; the response of the model to be evaluated is the text response output by the large language model; The response of the model to be evaluated is input into the basic reward model and the verifiable correctness signal reward model, respectively, to obtain the reward score output by the basic reward model and the reward score output by the verifiable correctness signal reward model. The basic reward model is a regression model trained based on human-annotated preference data. The final reward score for the response of the model to be evaluated is obtained by weighted summation based on the reward score output by each model and the weight of each model. The reward score output by the basic reward model is used to characterize human preference for the model response; the reward score output by the verifiable correctness signal reward model is used to characterize the correctness of the model response in a specific aspect. The verifiable correctness signal reward model includes: multiple verification agents; The step of inputting the response of the model to be evaluated into the verifiable correctness signal reward model to obtain the reward score output by the verifiable correctness signal reward model includes: The correctness of the model response to be evaluated is verified using a target agent, and a reward score is obtained from the output of the target agent; the target agent is a factual verification agent selected from the plurality of verification agents; the model response to be evaluated includes two different model responses; The step of using the target agent to verify the correctness of the response of the model to be evaluated, and obtaining the reward score output by the target agent, includes: The two different model responses are input into the factual verification agent to identify the differences between the two different model responses in the claimed facts, obtain the difference identification result, and generate the corresponding difference query based on the difference identification result; Evidence retrieval is performed using the difference query, and fact verification is conducted on the difference between the two different model responses based on the retrieved evidence. Each model response is scored according to the verification results to obtain a score for each model response. The evidence retrieval includes: retrieving evidence through an external search engine or by utilizing the knowledge base within a large language model. Based on the score of each model response, the reward score of the factual verification agent for the response of the model to be evaluated is determined; Among them, the model response that is factually correct scores higher than the model response that is factually incorrect.

2. The method according to claim 1, characterized in that, The step of inputting the response of the model to be evaluated into the basic reward model and the verifiable correctness signal reward model respectively to obtain the reward score output by each model includes: The response of the model to be evaluated is input into the basic reward model to obtain the human preference score for the response of the model to be evaluated, and the preference score is determined as the reward score output by the basic reward model.

3. The method according to claim 1, characterized in that, The verifiable correctness signal reward model also includes: a router; The target agent is obtained based on the following steps: Based on the response of the model to be evaluated, a corresponding verification command is generated and input into the router. The router then selects a verification proxy and filters out the target proxy to be used from the multiple verification proxies. The target agent includes at least one verification agent.

4. The method according to claim 3, characterized in that, The step of inputting the verification command into the router, selecting an verification proxy through the router, and filtering out the target proxy to be used from the multiple verification proxies includes: The verification command and the description information of each verification agent among the plurality of verification agents are input into the natural language model, and the natural language model selects the matching target agent from the plurality of verification agents according to the input command and the description information of each verification agent. The description information includes: the functionality and usage conditions of the verification agent.

5. The method according to claim 1 or 4, characterized in that, The target proxy is: the instruction that follows the verification proxy selected from the plurality of verification proxies; The step of using the target agent to verify the correctness of the response of the model to be evaluated, and obtaining the reward score output by the target agent, includes: Extract at least one constraint condition for the model response from the verification instruction, and generate a corresponding constraint verification script based on the constraint condition; The constraint verification script is used to perform constraint verification on the response of the model to be evaluated, to determine whether the response of the model to be evaluated satisfies each of the at least one constraint condition, and to obtain the constraint verification result. Calculate the average score of each constraint condition in the constraint verification results to obtain the reward score of the instruction compliance verification agent for the response of the model to be evaluated.

6. The method according to claim 1, characterized in that, The step of using the target agent to verify the correctness of the response of the model to be evaluated, and obtaining the reward score output by the target agent, includes: The reward score output by the target agent is obtained by weighted summing of the reward scores output by each of the multiple verification agents. The reward score output by the target agent is: the reward score output by the verifiable correctness signal reward model for the response of the model to be evaluated.

7. A reliable reward evaluation device that combines human preferences with verifiable correctness signals, characterized in that, The device includes: The acquisition module is used to acquire the response of the model to be evaluated; the response of the model to be evaluated is the text response output by the large language model. The reward scoring module is used to input the response of the model to be evaluated into the basic reward model and the verifiable correctness signal reward model, respectively, to obtain the reward score output by the basic reward model and the reward score output by the verifiable correctness signal reward model; the basic reward model is a regression model trained based on human-annotated preference data; The reward scoring module is also used to perform a weighted summation based on the reward score output by each model and the weight of each model to obtain the final reward score for the response of the model to be evaluated. The reward score output by the basic reward model is used to characterize human preference for the model response; the reward score output by the verifiable correctness signal reward model is used to characterize the correctness of the model response in a specific aspect. The verifiable correctness signal reward model includes: multiple verification agents; The reward scoring module is specifically used to verify the correctness of the response of the model to be evaluated using a target agent, and to obtain the reward score output by the target agent; the target agent is a factual verification agent selected from the plurality of verification agents; the response of the model to be evaluated includes two different model responses; The reward scoring module is further configured to input the two different model responses into the factual verification agent, identify the differences between the two different model responses in the claimed facts, obtain the difference identification result, and generate a corresponding difference query based on the difference identification result; The reward scoring module is further configured to use the difference query to retrieve evidence, verify the difference between the two different model responses based on the retrieved evidence, and score each of the two different model responses according to the verification results to obtain a score for each model response; the evidence retrieval includes: retrieving evidence through an external search engine or by utilizing the knowledge base within a large language model; The reward scoring module is further used to determine the reward score of the factual verification agent for the response of the model to be evaluated based on the score of each model response; Among them, the model response that is factually correct scores higher than the model response that is factually incorrect.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the reliable reward evaluation method combining human preferences and verifiable correctness signals as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the reliable reward evaluation method that combines human preferences with verifiable correctness signals as described in any one of claims 1 to 6.

Images