Training method and rewarding method of reasoning process evaluation model

Abstract

The invention discloses a training method and rewarding method for a reasoning process evaluation model, and the method comprises the steps: generating a plurality of reasoning contents tau for a given problem x through a large reasoning model, enabling each reasoning content tau to have a plurality of reasoning steps a, carrying out the marking of the correctness of each reasoning step a, and carrying out the rewarding of the reasoning content tau. Obtaining a plurality of data sets with correctness labels in the inference step a; and performing scoring training on each generated step a for a plurality of set problems x through a constructed correctness label in a reference data set with an expected evaluation function by utilizing a reasoning large model. According to the technology, the problem of complex information redundancy in the process inference model is effectively relieved by constructing the state evaluation function and the contrast loss function, meanwhile, modeling is conducted through the context dependency relationship between the steps, the distinguishing capacity of the process reward model for the correct steps and the wrong steps is improved, and the accuracy of the process reward model is improved. Therefore, the accuracy of a large model reasoning result is improved.

Description

Technical Field

[0001] This invention belongs to the field of model training technology, specifically relating to a training method and reward method for an evaluation model of reasoning process. Background Technology

[0002] Process reward models are primarily used to evaluate the quality of intermediate steps in the generation of large models, playing a crucial role in improving the inference results of large models. In complex problem scenarios, such as mathematical reasoning or multi-step decision-making, the accuracy and effectiveness of each intermediate step significantly impacts the overall output of the large model. Unlike outcome reward models that only focus on the final result, process reward models provide detailed feedback information at each stage of the large model's result generation process. By capturing the value of intermediate steps, process reward models can gain a deeper understanding of the rationality of each action generated by the large model, helping to output optimal results.

[0003] Existing technology and existing problems: (1) The reasoning steps generated by large language models (LLMs) contain a lot of information, and some redundant information may cause errors in the reasoning direction; (2) Existing methods typically define the process reward model as a classification problem, using cross-entropy loss to independently determine whether each intermediate step is correct. This method cannot fully utilize the contextual dependencies between steps, which may lead to classification errors. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a training method and reward method for a reasoning process evaluation model. The aim is to use a state evaluation model to score a given step, and the large model selects the best process to generate the answer based on the score, thereby guiding the model toward the correct answer.

[0005] The technical solution adopted in this invention is as follows: In the first aspect, the present invention provides a training method for a reasoning process evaluation model. First, a large reasoning model is used to generate several reasoning contents τ for a given problem x. Each reasoning content τ has several reasoning steps a. Then, the correctness of each reasoning step a is labeled, and a dataset of several reasoning steps a with correctness labels is obtained. Then, using a large inference model, for several set questions x, a scoring training is performed on each generated step a by constructing an evaluation function with expectations and referring to the correctness labels in the dataset.

[0006] In conjunction with the first aspect, the present invention provides a first embodiment of the first aspect, wherein the formula of the evaluation function is as follows: in, It is the inverse function of the sigmoid function. It is an exponential function. n This represents the number of steps to reach the final reasoning result. This represents a reasoning statement obtained from input question x. From the steps a t+1 arrive a n The entire process of obtaining the final structure; if the correct answer is obtained, The indicator function takes a value of 1, and the incorrect answer takes a value of 0.

[0007] In conjunction with the first aspect or the first embodiment of the first aspect, the present invention provides a second embodiment of the first aspect, which compares correct steps with incorrect steps using a loss function based on the evaluation value obtained by the evaluation function. The formula for the loss function is as follows: in, c Indicates the number of correct steps. w Indicates the number of erroneous steps. This is a marginal hyperparameter used to adjust the relative importance between correct and incorrect steps.

[0008] In conjunction with the second implementation of the first aspect, the present invention provides a third implementation of the first aspect, which uses a joint loss function to compare erroneous steps with correct steps. The formula for the joint loss function is as follows: in c Indicates the number of correct steps. w Indicates the number of erroneous steps.

[0009] In conjunction with the first aspect or the first embodiment of the first aspect, the present invention provides a fourth embodiment of the first aspect, which involves obtaining a plurality of reasoning steps with correctness labels. a When dealing with datasets, set a sliding window w for the same inference content. t Several steps that are related and satisfy the sliding window size w a Combined into a single step The step that was originally at the end in the logical progression order a The correctness label is used as a step in the merged process. The correctness label.

[0010] In conjunction with the first aspect or the first embodiment of the first aspect, the present invention provides a fourth embodiment of the first aspect, wherein each reasoning step is given aWhen verifying the correctness of all annotations, a reference model with a larger number of parameters is used to evaluate each inference step generated by the inference model. a All subsequent inference steps are regenerated from the reference large model. a And use regular expression matching to perform subsequent inference steps that are regenerated. a The conclusion is used as a reference, if the current reasoning step a If the regenerated conclusion is correct, then mark the reasoning steps. a If the conclusion is incorrect, then it is considered an incorrect step.

[0011] Secondly, the present invention also provides a reward method, which uses an evaluation model obtained by training any of the above-mentioned reasoning process evaluation models to evaluate each reasoning process generated by the large reasoning model. a A correctness assessment is performed, and the reasoning model selects the reasoning content with the highest average score as the answer to question x based on the assessment model.

[0012] The beneficial effects of this invention are as follows: (1) The technology of the present invention effectively alleviates the complex information redundancy problem in the process reasoning model by constructing a state evaluation function and a contrast loss function. At the same time, it uses the contextual dependency relationship between steps to model, which improves the ability of the process reward model to distinguish between correct steps and incorrect steps, thereby improving the accuracy of the reasoning results of the large model. (2) The state evaluation function designed in this invention can significantly reduce the score obtained when the first erroneous step occurs. Through the obvious difference in scores between steps, the erroneous step in the reasoning process can be accurately located, which facilitates the subsequent targeted tuning of the model and improves the efficiency of model optimization. (3) This invention utilizes large models with different parameter quantities to construct high-quality synthetic datasets. By automatically generating multi-trajectory inference data and combining it with external large models to label the correctness of the steps, and by using a sliding window to merge steps, it reduces the cost of manual labeling and reduces redundant information in the data, providing a fine-grained and highly reliable dataset for model training. (4) In this invention, a modified loss function is used to amplify the difference in evaluation scores of different step states through exponential and logarithmic operations, and marginal hyperparameters are introduced to adjust the relative importance of correct and incorrect steps, thereby strengthening the model’s ability to learn the differences between correct and incorrect steps and within incorrect steps, enabling the model to capture step value more accurately and improve the accuracy and robustness of the inference stage. Detailed Implementation

[0013] The present invention will be further explained below with reference to specific embodiments.

[0014] Example 1: This embodiment discloses a training method for a reasoning process evaluation model. First, a large reasoning model is used to generate several reasoning contents for a given problem x. Each reasoning content There are several reasoning steps 'a', and then each reasoning step is given... a All steps were labeled for correctness, and several reasoning steps with correctness labels were obtained. a The dataset; then, using a large inference model, for several given questions x, a constructed evaluation function with expectations is used to evaluate the correctness labels in the dataset for each generated step. a Conduct scoring training.

[0015] In this embodiment, a state evaluation function is specifically designed to measure the score of each step, indicating whether it is correct or incorrect. This score represents the probability of achieving the correct final answer given a state s-action a. A higher score indicates a greater likelihood that the action will guide the model to the correct answer. The score of this evaluation function increases with the number of consecutive correct steps and decreases with the number of incorrect steps. The state evaluation function is as follows: It is the inverse function of sigmoid. It is an indicator function, where n represents the number of steps to reach the final reasoning result. This indicates that input x follows the action steps. arrive a n The entire process of obtaining the final result is shown. If the correct answer is obtained, the indicator function is set to 1; otherwise, it is set to 0.

[0016] Furthermore, in this embodiment, in order to train the evaluation function, an exponential form of the loss function is introduced to amplify the differences in the results, thereby increasing the discriminative power. The specific method is as follows: I. Methods for constructing high-quality synthetic datasets This step proposes a synthetic data construction framework based on a large model, which generates high-quality, multi-granular training data through multi-trajectory sampling, step correctness verification, and sliding window merging techniques.

[0017] 1. Multi-trajectory sampling mechanism For a given problem x, N different reasoning trajectories are generated using a large reasoning model, i.e. Each of the trajectories t i It consists of n reasoning steps, that is ={ ， ,…， } Each step a j It represents the intermediate reasoning process from a question to an answer, such as formula derivation and logical judgment.

[0018] 2. Method for verifying the correctness of steps Unlike the large inference model used in the above steps, a reference large model with a larger number of parameters is used for evaluation.

[0019] Among them, for trajectory t i Each step in a j This process continues, generating subsequent content until the final answer is obtained. Then, regular expression matching or domain-specific rules are used to determine the correctness of the final answer. a j Once the correct answer is generated, mark it. a j This is the correct step (label=1). If an incorrect answer is generated, then... a j All subsequent steps are marked as errors (label=0).

[0020] 3. Use a sliding window to reduce the dataset size. First, initialize the sliding window size set. = ,in w i This represents different window sizes. Then, for the trajectory... Each position in j According to window size w i Merging steps a j arrive To form new steps The label for the new step is inherited from the last step in the original window. The labels are used to generate training datasets of various granularities by combining different window sizes.

[0021] II. Constructing and Training the State Evaluation Function 1. Construct the state evaluation function This embodiment proposes a state evaluation function based on probability estimation to measure the probability that each reasoning step will guide the model toward the correct answer.

[0022] Given a sequence of historical steps a 1:t-1 and current stepsa t The state evaluation function is defined as: in, It is the inverse function of the Sigmoid function (i.e., the Logit function), which maps probability values ​​to the real number domain; For indicator functions, if from step a t The initial value is 1 if the final answer generated is correct, and 0 otherwise. Is the model in a given historical step? a 1:t-1 The strategy for generating subsequent steps, and Take the expectation for all possible sequences of subsequent steps.

[0023] Indicates the execution steps a t Finally, the model obtains the log-odds ratio of the correct answer. The higher the score, the more likely the current step is to guide the model toward the correct answer.

[0024] This function has the following characteristics: consecutive correct steps accumulate positive rewards, causing the score to rise; incorrect steps will cause the score to fall; and it is more sensitive to errors in early critical steps.

[0025] 2. Employ a contrastive loss function to enhance the distinction between correct and incorrect steps. (1) Loss comparison between correct and incorrect steps The main loss function formula is as follows: Where c represents the number of correct steps and w represents the number of incorrect steps. This represents the state evaluation score for the t-th correct step. This represents the state evaluation score for the j-th erroneous step. It is a marginal hyperparameter, where This is used to adjust the relative importance of correct and incorrect steps.

[0026] In the loss function described above, exponential and logarithmic operations are used to amplify the state evaluation at different time steps. The gap between them. For each correct step ci, its state evaluation. It will be converted to exp( using the exponential function) This conversion method amplifies larger state assessment scores and relatively diminishes smaller state assessment scores, thus reinforcing the difference between the two.

[0027] Because the exponential function amplifies the score differences, the larger Amplified by the exponent, smaller It is further suppressed, while the log-likelihood form ensures probability normalization, marginal parameter Ensure that even if the score for incorrect steps is boosted, it remains significantly lower than that for correct steps. Specifically, during the calculation process, the score for incorrect steps... Add a positive scalar However, this increase is insufficient to change the relatively low score of incorrect steps compared to correct steps. Therefore, this approach effectively emphasizes the importance of correct steps while teaching the model how to distinguish between correct and incorrect steps.

[0028] (2) Internal comparison loss of the error step Refer to the following formula: This formula is designed to distinguish relatively better erroneous steps that are close to the correct inference path, forcing the model to learn the subtle differences between erroneous steps and improving the model's sensitivity to the quality of the inference path.

[0029] (3) Joint loss function III. A reward mechanism is formed for the inference model based on the evaluation model trained above. For a large-scale reasoning model, given an input question x, the model samples and generates m candidate answers, each containing n reasoning steps, i.e. { ， ,…， } .

[0030] Then the process reward model applies to each step of each answer. a i Calculate the state assessment score c i Calculate the overall score for each answer, i.e. Finally, the result with the highest overall score is selected as the output of question x.

[0031] Furthermore, based on the above method, this embodiment provides several specific problems x for explanation.

[0032] Example 1: The Chicken and Rabbit Problem 1. Data generation and annotation Generate 20 inference trajectories using a large model with a small number of parameters (such as GPT-3), each trajectory containing 3-5 steps. For example: Track A: Assume all are chickens → Count legs: 40 → 16 legs are missing → Each time a rabbit is replaced, 2 more legs are added → Count 8 rabbits and 12 chickens. Track B: Let there be x chickens and y rabbits → Set up the equation x + y = 20 → Set up the equation 2x + 4y = 56 → Solve the simultaneous equations → We get x = 12, y = 8 Trajectory C: Directly guess 10 chickens → Calculate 60 legs → Error found → Re-assume 15 chickens → Calculate 50 legs → Continue adjusting Then, use an external model with a larger number of parameters (such as GPT-4) to label each step: if the step assumes that all the animals are chickens and can be logically deduced to obtain the correct answer, it is labeled as "correct"; if the trajectory C directly guesses that there are 10 chickens and the subsequent steps are repeatedly adjusted and not solved correctly, it is labeled as "incorrect".

[0033] 2. Sliding window preprocessing Set the sliding window size w=2. Combine the first two steps of trajectory A (assuming all are chickens) → calculate the number of legs (40) into a composite step A1, with the label remaining correct. Combine the direct guess of 10 chickens → calculate the number of legs (60) of trajectory C into a composite step C1, with the label incorrect. A total of 500 merged training samples are generated, with 60% of the steps being correct and 40% being incorrect.

[0034] 3. Model Training The state evaluation function calculates that for step B, with x chickens and y rabbits, the probability of generating the correct answer in subsequent steps is 0.9; for step C, the probability of directly guessing 10 chickens is 0.2.

[0035] Loss function optimization: setting hyperparameters =0.1, in In the study, the scores of correct steps, after being amplified, were significantly higher than those of incorrect steps, thus reinforcing the contrast. The erroneous steps in trajectory C are sorted by score to distinguish between blind guessing and adjusted assumptions.

[0036] 4. Application in the reasoning stage The 10 newly generated trajectories were scored. Trajectory B had an average score of 0.85, Trajectory A had 0.78, and Trajectory C had 0.42. Trajectory B was chosen as the output because its steps were logically coherent and it had the highest score.

[0037] Example 2: Solving a linear equation in one variable The input equation is 2x + 5 = 13. Find x.

[0038] 1. Data generation and annotation The reasoning process is as follows: Locus A: Rearranging terms gives 2x = 13 - 5 → Calculating the right side gives 8 → Dividing both sides by 2 gives x = 4 Locus B: Subtracting 5 from both sides gives 2x = 8 → directly derive x = 4 Locus C: Adding 5 to both sides gives 2x = 18 → Calculation error, x = 9 The steps for labeling trajectories A and B on the external model were correct, but adding 5 steps to both sides of trajectory C resulted in an error, and the labeling was incorrect.

[0039] 2. Sliding window preprocessing Using a window size w=3, the three steps of trajectory A are merged into one composite step, and the label is correct; the first two steps of trajectory C are merged into a composite step with 5 added to both sides → calculation error, and the label is incorrect.

[0040] 3. Model Training The state evaluation function scores 0.95 for the step of subtracting 5” on both sides of trajectory E and 0.1 for the step of adding 5” on both sides of trajectory F.

[0041] In the loss function, the exponent term of the correct step, where exp(0.95)=2.58 for the correct step, shows a significant difference from exp(0.1+0.1)=1.22 for the incorrect step. The model adjusts the parameters through backpropagation to strengthen the correct judgment of the "transition direction".

[0042] 4. Application in the reasoning stage Fifteen trajectories were generated. Trajectory B had an average score of 0.92, trajectory A 0.88, and trajectory C 0.21. Trajectory E was chosen as the answer because it was the simplest to execute and had the highest score, avoiding redundant calculations.

[0043] In the examples above, 10-20 trajectories are generated for each problem. The external model annotation accuracy is approximately 85%, the window size w∈{2,3,4}, and samples of different granularities are mixed to reduce redundancy by approximately 30%. Using the Adam optimizer with a learning rate of 1e-4, after 50 epochs of training, the average score for correct steps improves from 0.5 to 0.85, while the average score for incorrect steps decreases from 0.5 to below 0.2. In a test of 100 math problems, this technique improved the inference accuracy of the large model from 60% to 88%, and the accuracy of locating incorrect steps reached 92%.

[0044] This invention is not limited to the optional embodiments described above, and anyone can derive other various forms of products based on the inspiration of this invention. The specific embodiments described above should not be construed as limiting the scope of protection of this invention; the scope of protection of this invention should be determined by the claims, and the specification can be used to interpret the claims.

Claims

1. A training method for a reasoning process evaluation model, characterized in that: First, a large-scale reasoning model is used to generate several reasoning contents for a given problem x. Each reasoning content There are several reasoning steps 'a', and then each reasoning step is given... a All steps were labeled for correctness, and several reasoning steps with correctness labels were obtained. a The dataset; Then, using a large-scale inference model, for several defined questions x, a constructed evaluation function with expectations is used to evaluate the correctness labels in the dataset for each generated step. a Conduct scoring training.

2. The training method for a reasoning process evaluation model according to claim 1, characterized in that: The formula for the evaluation function is as follows: ； in, It is the inverse function of the sigmoid function. It is an exponential function. n This represents the number of steps to reach the final reasoning result. This represents a reasoning statement obtained from input question x. From the steps a t+1 arrive a n The entire process of obtaining the final structure; if the correct answer is obtained, The indicator function takes a value of 1, and the incorrect answer takes a value of 0.

3. The training method for a reasoning process evaluation model according to claim 1 or 2, characterized in that: Based on the evaluation value obtained from the evaluation function, a loss function is used to compare the correct and incorrect steps. The formula for the loss function is as follows: ； in, c Indicates the number of correct steps. w Indicates the number of erroneous steps. This is a marginal hyperparameter used to adjust the relative importance between correct and incorrect steps.

4. The training method for a reasoning process evaluation model according to claim 3, characterized in that: A joint loss function is used to compare erroneous and correct steps. The formula for the joint loss function is as follows: ； Among them, the loss function The formula is as follows: ； in c Indicates the number of correct steps. w Indicates the number of erroneous steps.

5. A training method for a reasoning process evaluation model according to claim 1 or 2, characterized in that: In obtaining several reasoning steps labeled with correctness a When dealing with datasets, set a sliding window w for the same inference content. τ Several steps that are related and satisfy the sliding window size w a Combined into a single step The step that was originally at the end of the logical progression order a The correctness label is used as a step in the merged process. The correctness label.

6. A training method for a reasoning process evaluation model according to claim 1 or 2, characterized in that: In each reasoning step a When verifying the correctness of all annotations, a reference model with a larger number of parameters is used to evaluate each inference step generated by the inference model. a All subsequent inference steps are regenerated from the reference large model. a And use regular expression matching to perform subsequent inference steps that are regenerated. a The conclusion is used as a reference, if the current reasoning step a If the regenerated conclusion is correct, then mark the reasoning steps. a If the conclusion is incorrect, then it is considered an incorrect step.

7. A reward method, characterized in that: The evaluation model obtained by using the training method of the reasoning process evaluation model of claim 1 or 2 is used for each reasoning process generated by the large reasoning model. a A correctness assessment is performed, and the reasoning model selects the reasoning content with the highest average score as the answer to question x based on the assessment model.

Images