Large language model long thinking chain verification method and device based on reasoning process abstract

Abstract

The invention relates to a large language model long thinking chain verification method and device based on an inference process abstract. The method comprises the following steps: obtaining a to-be-verified inference thinking chain; abstracting the obtained reasoning thinking chain to obtain a linear path containing a key reasoning link; and gradually executing verification on the linear path by adopting a checker, judging whether each step is correct or not, positioning the first wrong step, regarding the subsequent steps as invalid, and not independently verifying. Compared with the prior art, the method has the advantages that an accurate and efficient verification scheme is provided, the preciseness of process verification is guaranteed, the verification and labeling cost is remarkably reduced, and large-scale data set construction and model training can be supported.

Description

Technical Field

[0001] This invention relates to the field of process verification technology for large language models, and in particular to a method and apparatus for verifying long thought chains of large language models based on reasoning process summaries. Background Technology

[0002] Currently, large language models rely on long thought chains to handle complex mathematical reasoning tasks, that is, to generate lengthy and complicated thought processes before arriving at a final conclusion. However, large language models often generate unreliable intermediate processes, which greatly hinders their application in the real world.

[0003] Process validation of large language models refers to verifying the correctness of a series of intermediate reasoning steps generated by the large language model and providing error localization and feedback. Process validation is an effective supplement to result validation. For thought chains with correct results, process validation can detect "false positives," i.e., situations where the correct answer is obtained by chance through logical jumps, erroneous reasoning, invalid proofs, or guessing. For thought chains with incorrect results, process validation can effectively analyze the causes of errors. In summary, process validation can provide technical support for fine-grained supervised learning and capability assessment of large language models.

[0004] Existing process validators such as CoT Monitor and Heimdall are based on large language models as validator foundations, generating natural language verification trajectories for thought chains and providing correctness judgment and error analysis.

[0005] (a) CoT Monitor employs cue word engineering, which allows a language model to verify the complete thought process and output code of a programming model. This can effectively detect the phenomenon of "reward hacking," where a programming model passes unit tests through fraudulent means.

[0006] (b) Heimdall discards the internal thought process and sets the validation object as the native summary part of the long inference model. Although Heimdall can provide process validation, its training labels and evaluation metrics mainly come from result validation, that is, the correctness of the final result of the content to be validated.

[0007] The shortcomings of existing technologies are mainly reflected in the following aspects: 1) There is no faithful and easily verifiable thought chain quality proxy. The overhead of full thought chain verification is very large, and simply selecting the summary part provided by the large language model will lead to excessive omission and cause the verifier to misjudge; 2) There is a lack of high-quality fine-grained labels. Existing process verifiers rely on heuristic methods such as cue word engineering, manually designed criteria, result verification, or Monte Carlo sampling. These methods introduce a lot of noise, resulting in process verifiers with limited performance, which cannot be generalized to high-performance language models to solve high-difficulty reasoning tasks. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art, which has a very high overhead for full thought chain verification and the shortcomings of simply selecting the summary part of the large language model itself, which leads to excessive omission and misjudgment by the verifier. Therefore, this invention provides a method and device for long thought chain verification of large language models based on the summary of the reasoning process.

[0009] The objective of this invention can be achieved through the following technical solutions:

[0010] A method for validating long thought chains in large language models based on reasoning process summaries includes the following steps:

[0011] Obtain the reasoning chain to be verified;

[0012] The obtained reasoning chain is summarized to obtain a linear path containing key reasoning steps;

[0013] The verification process is performed step by step along the linear path to determine whether each step is correct. The first incorrect step is located, and subsequent steps are considered invalid and are no longer verified separately.

[0014] Furthermore, the training process of the tester includes:

[0015] S301: Collect questions, generate multiple reasoning thought chains for training, and perform summarization processing to obtain summary paths, forming an unlabeled pool;

[0016] S302: The tester to be trained is used to independently verify the summary path in the unlabeled pool multiple times to obtain multiple samples of error location prediction and interpretation;

[0017] S303: Select samples with significant prediction discrepancies from the obtained samples as high uncertainty samples;

[0018] S304: The high-uncertainty samples are labeled by experts using the expert method to obtain high-confidence labels and explanations, which constitute the labeled set;

[0019] S305: Only samples consistent with the expert annotations are retained for supervised learning, and a fine-grained reward function is used to optimize the parameters of the tester;

[0020] S306: Repeat steps S302-S305 for multiple iterations until the preset iteration cutoff condition is met.

[0021] Furthermore, the expert method performs expert annotation according to an annotation protocol, which includes:

[0022] Task requirements: Based on the problem and the corresponding summary path, identify the earliest step where the error occurred and provide a natural language explanation. If there are no errors, mark it as correct.

[0023] Fuzzy processing: Define steps that explicitly violate logic, mathematics, or facts as errors, and all others as correct;

[0024] Consistency mechanism: Each sample is independently labeled by multiple experts, and the final labeling result is adopted when the labeling results of multiple experts meet the consistency condition.

[0025] Reference support: Provide standard answers or reference solutions to experts.

[0026] Furthermore, the consistency conditions include:

[0027] All N experts judged it as correct, where N is the total number of experts;

[0028] At least N-1 experts are completely in agreement on the wrong point;

[0029] Although the experts disagreed, the errors were concentrated in the range of two adjacent steps.

[0030] Furthermore, the annotation protocol also includes:

[0031] Explanation Requirements: Experts must explain the reasons for their judgments, which the testing equipment will then learn from.

[0032] Furthermore, the fine-grained reward function is an exponentially decaying reward function, the reward value of which decays as the step distance between the tester's prediction result and the expert's annotation result increases.

[0033] Furthermore, the method employs a large language model to summarize the acquired reasoning thought chain.

[0034] Furthermore, the summarization process specifically includes:

[0035] A large language model is used to extract the core reasoning path from the acquired reasoning thought chain, and then the core reasoning path is divided into steps to obtain a more uniform step structure.

[0036] Furthermore, before the summarization process, the acquired reasoning chain is combined with preset prompt words, and then processed through a large language model.

[0037] The present invention also provides a long thought chain verification device for a large language model based on reasoning process summary, including a memory and a processor. The memory stores a computer program, and the processor calls the computer program to execute the steps of the method described above.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] (1) This invention first uses a large language model to summarize the long thought chain, compressing the complex and redundant reasoning process into a linear path containing key reasoning steps. This summarized path faithfully preserves the core logic supporting the final answer, eliminating trial and error, redundancy, and branch exploration. Subsequently, the validator performs verification step by step on this summarized path, judging whether each step is correct and locating the first erroneous step. Once the earliest error is found, subsequent steps are considered "invalid" and no longer verified separately. This design ensures the rigor of process verification while significantly reducing the cost of verification and annotation, enabling it to support large-scale dataset construction and model training, and proposes a verification scheme that is both accurate and efficient.

[0040] (2) This invention establishes a strict expert annotation protocol to guide the manual review and judgment of abstract paths, realize the large-scale introduction of expert annotation, and generate a large amount of high-quality supervision data.

[0041] (3) This invention designs an iterative framework driven by active learning, which efficiently utilizes expert annotations and optimizes the performance of the verifier in a targeted manner. Attached Figure Description

[0042] Figure 1 This is a schematic diagram illustrating the difference between a verification paradigm provided in this embodiment of the invention and traditional result verification and process verification;

[0043] Figure 2 This is a schematic diagram of a long thought chain verification method for a large language model based on reasoning process summarization provided in an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0045] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0046] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0047] Example 1

[0048] The purpose of this embodiment is to propose a verification paradigm that balances accuracy and efficiency by designing a faithful and easily verifiable thought chain quality proxy, enabling large-scale expert annotation, and thus obtaining a high-performance process verifier. The differences between the verification paradigm proposed in this embodiment and traditional result verification and process verification are as follows: Figure 1 As shown.

[0049] Specifically, such as Figure 2 As shown, a method for validating long thought chains in large language models based on reasoning process summaries is provided, including the following steps:

[0050] S1: Obtain the reasoning chain to be verified;

[0051] S2: Summarize the obtained reasoning chain to obtain a linear path containing key reasoning steps;

[0052] S3: Use a checker to perform verification step by step on the linear path, determine whether each step is correct, locate the first incorrect step, and treat subsequent steps as invalid and no longer verify them separately.

[0053] Preferably, the method uses a large language model to summarize the acquired reasoning thought chain.

[0054] Essentially, this solution first uses a large language model to summarize the long thought process, compressing the complex and redundant reasoning process into a linear path containing key reasoning steps. This summarized path faithfully preserves the core logic supporting the final answer, eliminating trial and error, redundancy, and branching exploration.

[0055] The summarization process involves two steps: first, the large language model extracts the core reasoning path using designed prompts; then, it breaks down the reasoning path into steps, resulting in a more even and clear structure. Subsequently, the validator performs verification step-by-step on this summarized path, determining the correctness of each step and identifying the first erroneous step. Once the earliest error is detected, subsequent steps are considered "invalid" and no longer validated individually.

[0056] This design ensures the rigor of the process verification while significantly reducing the cost of verification and annotation, enabling it to support the construction of large-scale datasets and model training.

[0057] Preferably, the training process of the tester includes:

[0058] S301: Collect questions, generate multiple reasoning thought chains for training, and perform summarization processing to obtain summary paths, forming an unlabeled pool;

[0059] That is, initial preparation:

[0060] Questions are collected from multiple question banks to generate multiple reasoning thought chains, and then summarized using a general model to form a large-scale unlabeled pool.

[0061] Use an existing open-source dataset to initialize a limited-capacity validator as a starting point.

[0062] S302: The tester to be trained is used to independently verify the summary path in the unlabeled pool multiple times to obtain multiple samples of error location prediction and interpretation;

[0063] S303: Select samples with significant prediction discrepancies from the obtained samples as high uncertainty samples;

[0064] S304: The high-uncertainty samples are labeled by experts using the expert method to obtain high-confidence labels and explanations, which constitute the labeled set;

[0065] S305: Only samples consistent with the expert annotations are retained for supervised learning, and a fine-grained reward function is used to optimize the parameters of the tester;

[0066] S306: Repeat steps S302-S305 for multiple iterations until the preset iteration cutoff condition is met.

[0067] That is, the iterative process (each round contains four steps):

[0068] Verification reasoning: The current version of the validator performs multiple independent verifications on the summary paths in the unlabeled pool, resulting in multiple error location predictions and explanations.

[0069] Uncertainty assessment: By using statistical consistency indicators, samples with large prediction discrepancies are selected, and these are considered to be the "most difficult to judge and most valuable" samples.

[0070] Expert annotation: These high-uncertainty samples were submitted to experts for annotation to obtain high-confidence labels and explanations.

[0071] Model update: The validator is updated using a dual training strategy, combining the existing labeled set.

[0072] Rejection-based fine-tuning: Only validation trajectories consistent with expert annotations are retained for supervised learning, while inconsistent samples are filtered out to ensure stable training.

[0073] Reinforcement learning optimization: A fine-grained reward function is designed to enable the validator to gradually learn more accurate verification capabilities while maximizing the utilization of expert knowledge. Specifically, the fine-grained reward function is an exponentially decaying reward function, where the reward decreases as the step distance between the validator's prediction result and the expert's annotation result increases; it can provide fine-grained rewards based on prediction errors to alleviate the sparsity problem of traditional reward functions.

[0074] An example of the form of this reward function is as follows:

[0075] Given the result of the validator prediction and expert annotation results * Define the exponentially decaying reward function as:

[0076]

[0077] Where λ∈(0,1) represents the degree of penalty for controlling location errors, and otherwise represents other values.

[0078] Multiple iterations and expansions:

[0079] As the rounds progress, the validator's performance gradually improves, enabling it to provide stable predictions on more unlabeled data. Simultaneously, the scale of expert annotations continuously expands, ultimately forming a large-scale, high-quality training dataset. After multiple iterations, the validator achieves near-expert-level accuracy and robustness.

[0080] Preferably, the expert method performs expert annotation according to an annotation protocol, the annotation protocol including:

[0081] 1. Task Requirements: Based on the problem and the corresponding summary path, identify the earliest step where the error occurred and provide a natural language explanation. If there are no errors, mark it as correct.

[0082] That is, experts need to identify the earliest step in the reasoning process based on the question and summary, and provide a brief explanation in natural language. If the reasoning is completely correct, it is explicitly marked as "correct".

[0083] 2. Fuzzy processing: Define steps that explicitly violate logic, mathematics, or facts as errors, and all others as correct;

[0084] That is, for steps that are “minor oversights but easily correctable”, the labeling protocol stipulates that they are not considered errors, but are checked until a step that clearly violates logic, mathematics or facts is encountered.

[0085] 3. Consistency Mechanism: Each sample is independently labeled by multiple experts, and the final labeling result is obtained when the labeling results of multiple experts meet the consistency condition.

[0086] The consistency conditions include:

[0087] All N experts judged it as correct, where N is the total number of experts;

[0088] At least N-1 experts are completely in agreement on the wrong point;

[0089] Although the experts disagreed, the errors were concentrated in the range of two adjacent steps.

[0090] Results annotated independently by three experts will only be accepted under the following circumstances:

[0091] All three experts determined it to be correct;

[0092] At least two experts are completely in agreement on the wrong point;

[0093] Although the three experts disagreed, the errors were concentrated in the range of two adjacent steps.

[0094] This approach avoids including low-confidence labels in the training set while ensuring the tolerance and high confidence of multiple expert opinions.

[0095] 4. Reference Support: Provide standard answers or reference solutions to experts to help quickly determine key steps.

[0096] 5. Explanation Requirements: Experts explain the reasons for their judgments, which are then used by the testing equipment to learn from these reasons.

[0097] In other words, experts need to provide brief explanations and reasons for their judgments, so that the model can not only learn to "judge right or wrong", but also learn "how to explain".

[0098] This protocol enables the construction of datasets that are both highly reliable and consistent, laying the foundation for subsequent model iteration training.

[0099] This embodiment also provides the following experimental verification process:

[0100] We trained a long thought chain checker based on the Qwen2.5-32B model and tested the performance of the long thought chain checker under different feedback granularities, proving the effectiveness of our training scheme.

[0101] We used a test set randomly selected from data unanimously approved by human experts to test the performance of the long thought chain validator at various granularities. The prediction accuracy ACC and F1 were used as indicators, and the results were compared with the current state-of-the-art open-source models Deepseek-R1-671B, Qwen3-235B and GPT-OSS-120B, as shown in Table 1.

[0102] Here, coarse-grained and fine-grained represent the model's performance in evaluating the overall correctness / incorrectness of the solution, and the model's performance in evaluating the specific erroneous steps in the solution, respectively.

[0103] Table 1

[0104]

[0105]

[0106] It can be observed that after using this self-iterative training method, the trained model achieved the highest detection accuracy at different detection granularities, surpassing the current models with the strongest overall performance. It achieved a prediction accuracy of 93.05% and an F1 score of 86.90% at the coarse feedback granularity, and a prediction accuracy of 81.61% and an F1 score of 71.46% at the fine feedback granularity.

[0107] It should be noted that this invention adopts a summary + step-by-step verification mode, and uses an active learning-driven human-computer interactive iterative training framework, namely: a loop of: verifier verification - selection of uncertain samples - expert annotation - updating the verifier. Theoretically, the above verification mode and verifier training scheme are all modifications of this invention.

[0108] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A large language model long thinking chain verification method based on an inference process summary, characterized in that, The method comprises the following steps: obtaining a reasoning thought chain to be verified; performing abstract processing on the obtained reasoning thought chain to obtain a linear path containing key reasoning links; using a verifier to perform verification step by step on the linear path, judging whether each step is correct, locating the first erroneous step, and regarding subsequent steps as invalid and not verifying them separately.

2. The large language model long thinking chain verification method based on inference process summary according to claim 1, characterized in that, The training process of the verifier comprises: S301: collecting questions, generating a plurality of reasoning thought chains for training, and performing abstract processing to obtain abstract paths, forming an unlabeled pool; S302: using the verifier to be trained to perform multiple independent verifications on the abstract paths in the unlabeled pool to obtain a plurality of error location prediction and explanation samples; S303: selecting samples with greater prediction divergence from the obtained samples as high-uncertainty samples; S304: using an expert method to perform expert labeling on the high-uncertainty samples to obtain high-confidence labels and explanations, forming a labeled set; S305: only retaining samples consistent with the expert labeling for supervised learning, and using a fine-grained reward function to optimize the parameters of the verifier; S306: repeatedly performing steps S302-S305 for multiple iterations until a preset iteration termination condition is met.

3. The method of claim 2, wherein the method is characterized by, The expert method performs expert labeling according to a labeling protocol, and the labeling protocol comprises: task requirements: based on the question and the corresponding abstract path, pointing out the earliest erroneous step and giving the corresponding natural language explanation, and if there is no error, marking it as correct; fuzzy processing: defining steps that clearly violate logic, mathematics or facts as errors, and the rest as correct; consistency mechanism: each sample is independently labeled by multiple experts, and when the labeling results of multiple experts meet the consistency condition, it is regarded as the final labeling result; reference support: providing standard answers or reference solutions to experts.

4. The large language model long thinking chain verification method based on inference process summary according to claim 3, characterized in that, The consistency condition comprises: N experts all determine that it is correct, and N is the total number of experts; at least N-1 experts are completely consistent in the error position; although there are differences among experts, the error positions are concentrated within two adjacent steps.

5. The large language model long thinking chain verification method based on inference process summary according to claim 3, characterized in that, The labeling protocol further comprises: explanation requirements: experts explain the reasons for the determination, which are used by the verifier to learn the reasons for the determination.

6. The large language model long thinking chain verification method based on inference process summary according to claim 2, characterized in that, The fine-grained reward function is an exponential decay reward function, and the reward value of the reward function decays with the increase of the step distance between the prediction result of the verifier and the labeling result of the expert.

7. The method of claim 1, wherein the method is characterized by, The method uses a large language model to perform abstract processing on the obtained reasoning thought chain.

8. The large language model long thinking chain verification method based on inference process summary according to claim 7, characterized in that, The abstract processing specifically comprises: using a large language model to extract a core reasoning path from the obtained reasoning thought chain, and then performing step division on the core reasoning path to obtain a more uniform step structure.

9. The large language model long thinking chain verification method based on an inference process summary according to claim 8, characterized in that, Before the abstract processing, the obtained reasoning thought chain is combined with a preset prompt word, and then processed by a large language model.

10. A large language model long thinking chain verification device based on an inference process summary, characterized by, The device comprises a memory and a processor, the memory stores a computer program, and the processor invokes the computer program to execute the steps of the method according to any one of claims 1-9. The device comprises a memory and a processor, the memory stores a computer program, and the processor invokes the computer program to execute the steps of the method according to any one of claims 1-9.

Images