Design method, system and device of large model reasoning path verifier based on preference tree

Abstract

The invention discloses a method, a system and a device for designing a large model reasoning path verifier based on a preference tree. Aiming at the application of a large language model in a complex reasoning task, the invention provides a novel reasoning path verifier, namely a preference tree-based verifier. According to the method, a preference tree is constructed through a best-first search algorithm, and step-level pairing data is collected to perform preference learning training on a verifier, so that the problem that an existing verifier is only trained on a binary annotation reasoning path and cannot fully utilize a relative relationship of intermediate steps is solved. Compared with a traditional binary classification training method, the verifier carefully captures nuisance between reasoning steps through step-level preference, and therefore the whole reasoning path is evaluated more accurately. The method effectively improves the accuracy and reliability of the verifier, especially has excellent performance in arithmetic and common sense reasoning tasks, and can be further used for assisting math teaching and other tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a design method, system and device for a large model reasoning path verifier based on a preference tree. Background Art

[0002] The reasoning ability of large language models is an important foundation for achieving efficient human-computer interaction. As these models develop, they have demonstrated excellent reasoning ability in solving complex problems. This ability not only includes simple questions and answers, but also covers complex logical reasoning, multi-step reasoning, and cross-domain knowledge integration. For example, in the fields of legal consultation, medical diagnosis, and technical support, the model can not only provide accurate information, but also reason based on the context and give professional advice. This makes the large language model not only a provider of information, but also an expert in problem solving.

[0003] However, current large language models still have some limitations when generating reasoning paths, such as the generated content may contain factual errors. In order to improve the reliability of the reasoning path, it is usually necessary to select the best one from multiple generated candidate paths according to specific rules. These rules include coherence, logical consistency, and consistency with known facts. Recent studies have introduced auxiliary models, called verifiers, to evaluate the quality of these candidate paths. These verifiers treat this task as a binary classification task, annotate the reasoning paths based on standard answers, and use these paths as training data for supervised learning, and then score the correctness of the candidate paths during testing. Furthermore, according to the granularity of the supervision signal, it can be divided into result supervision and process supervision. Result supervision provides instance-level feedback signals based on the correctness of the reasoning path results, and process supervision annotates the correctness of each reasoning step to provide step-level supervision signals.

[0004] The main problem with existing verifiers is that their optimization objectives are not completely consistent with the actual task. The verifier needs to accurately rank candidate paths, while the signal provided by the binary classification label is too coarse to capture the relative merits of different paths. Such supervisory signals are not sufficient to provide detailed feedback, limiting the potential for further improvement of the verifier.

[0005] On the other hand, the labeled data used in existing training methods often contain a certain degree of noise. Even if the final answer is correct, the reasoning process may not be completely accurate. These factors make existing binary classification methods susceptible to noisy labels, limiting the ability of the verifier to accurately verify the step. Summary of the invention

[0006] In order to solve the problem that it is difficult for a binary classification training verifier to obtain detailed supervision signals from its training mode, the present invention proposes a design method for a large model inference path verifier based on a preference tree, which can further improve the accuracy of evaluating candidate paths after learning the relative relationship between different inference paths.

[0007] The present invention solves the technical problem by adopting the following technical solution: a design method of a large model reasoning path verifier based on a preference tree, the method comprising the following steps:

[0008] (1) Using a large language model, we sample several subsequent reasoning paths starting from the reasoning step to be evaluated, and quantify the quality of the reasoning step based on the proportion of correct paths in the sampled reasoning paths to obtain the value of the reasoning step;

[0009] (2) Starting from the original question stem as the root node, a preference tree is constructed, where each node represents a reasoning step. Each iteration starts from the root node of the tree and includes the following sub-steps:

[0010] (2.1) Calculate the value of each child node under the current node and select the node with the highest value for further expansion;

[0011] (2.2) After selecting the highest value node, the preference tree is expanded by generating new child nodes; a large language model is used to sample several potential subsequent reasoning steps, each of which becomes a new child node of the previously selected highest value node; if the selected node represents the last step, the generation of an inference path is completed, and the expansion stage is omitted, and the current iteration ends; the next iteration starts again from the root node of the tree, and all nodes generated in the previous iteration do not participate in the node comparison of the current iteration;

[0012] (3) Create a preference pairing data training set containing the preference of the reasoning path from the preference tree obtained in step (2), including the following sub-steps:

[0013] (3.1) Trace each reasoning path from the root node to the leaf node in the preference tree, compare the value of each child node of each node, and combine these comparison results in pairs to construct a preference pairing data training set;

[0014] (3.2) Screen out the paired data with a value difference greater than a threshold in the preferred paired data training set;

[0015] (4) training a verifier, wherein the verifier is based on a large language model and adds a randomly initialized linear layer, which outputs a scalar value; and performing preference learning training on the verifier using the preference pairing data training set obtained in step (3);

[0016] (5) Use the target large language model to generate several reasoning paths for the question, use these candidate paths as input to the verifier, and let the verifier score them in turn, and select the path with the highest score as the final reasoning path.

[0017] Furthermore, step (1) is specifically as follows: the i-th reasoning step is denoted as y i , using a large language model from y i Start sampling N subsequent reasoning paths, and the set of sampled reasoning paths is recorded as Reasoning step y i The value of The calculation formula is as follows:

[0018]

[0019] Among them, a[P i j ] represents the jth reasoning path P i j The result, j = 1, ..., N, g represents the correct answer, 1 [a [P i j ]=g] means that if the jth reasoning path P i j If the result is correct, it takes 1, otherwise it takes 0.

[0020] Furthermore, in step (1), the potential subsequent reasoning steps for sampling are as follows: after determining the highest value node, the large language model π θ Sample k potential subsequent reasoning steps x represents the original question stem, y 1:i Represents the 1st to the i-th reasoning steps.

[0021] Furthermore, in step (3.1), the construction of the preference pairing data training set is specifically as follows:

[0022] Each entry in the preference pairing data training set is in the form of {(x,y + ,y - )}, where x represents the original question stem, y + is the reasoning path as a positive example, y - is the reasoning path as a negative example;

[0023] Each node in the preference tree is compared with the nodes at the same level in pairs; the nodes at the same level are various possible subsequent reasoning steps branching from the same previous context; according to the node value obtained in step (2.1), the relative relationship between these reasoning steps is obtained to form a preference pair; for the better step in each preference pair By selecting a correct reasoning path from the set of subsequent reasoning paths sampled in step (1), we can form a reasoning path y + ; For less optimal steps Select a reasoning path with an incorrect result from its potential subsequent reasoning path set to form a reasoning path y - .

[0024] Furthermore, in step (3.2), if and If the value difference between them is greater than the preset threshold α, the pairing data {(x,y + ,y - )}.

[0025] Furthermore, in step (4), a step-level ranking loss function is used to train the verifier. This loss function is designed to promote the selection of the most effective reasoning path. The formula is as follows:

[0026]

[0027] Where d is the step subscript where the two reasoning paths diverge, n is the subscript of the last reasoning step, σ(·) represents the sigmoid activation function, and r φ (x,y 1:i ) represents the verifier’s response to the input x and the inference step sequence {y1,…,y i}The output score is given by φ.

[0028] Furthermore, in step (5), for a given input x, M candidate reasoning paths {y (1) ,y (2) ,…,y (M)}, these reasoning paths are then ranked by the trained verifier, and the one with the highest score is selected as the most credible reasoning path.

[0029] The present invention also provides a design system of a large model reasoning path verifier based on a preference tree, the system comprising the following modules:

[0030] Value function design module: Using a large language model, we sample several subsequent reasoning paths starting from the reasoning step to be evaluated, and quantify the quality of the reasoning step based on the proportion of correct paths in the sampled reasoning paths to obtain the value of the reasoning step;

[0031] Preference tree construction module: Starting from the original question stem as the root node, the preference tree is constructed. Each node represents a reasoning step. Each round of iteration starts from the root node of the tree, including: calculating the value of each child node under the current node, selecting the node with the highest value, and expanding the preference tree by generating new child nodes; using a large language model to sample several potential subsequent reasoning steps, each of which becomes a new child node of the previously selected node with the highest value; if the selected node represents the last step, the generation of a reasoning path ends, and the expansion stage is omitted, and the current round of iteration ends; the next round of iteration starts again from the root node of the tree, and all nodes generated in the previous round of iteration do not participate in the node comparison of this round of iteration;

[0032] Preference pairing data training set construction module: trace each reasoning path from the root node to the leaf node in the preference tree, compare the value of each child node of each node, and combine these comparison results in pairs to construct the preference pairing data training set; filter out the pairing data in the training set whose value difference is greater than the threshold;

[0033] Validator training module: The validator is based on a large language model and adds a linear layer with randomly initialized output scalar values. The validator is trained for preference learning using the preference pairing data training set.

[0034] Best reasoning path selection module: Use the target large language model to generate several reasoning paths for the problem, use these candidate paths as verifier input, let the verifier score them in turn, and select the path with the highest score as the final reasoning path.

[0035] The present invention also provides a design device for a large model reasoning path verifier based on a preference tree, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the above-mentioned design method for the large model reasoning path verifier based on the preference tree.

[0036] The present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned design method of the large model reasoning path verifier based on the preference tree is implemented.

[0037] The beneficial effects of the present invention are as follows:

[0038] 1. The method of the present invention proposes to train a large model reasoning path verifier by using preference learning, which makes up for the problem that the binary classification training method cannot provide a fine supervision signal;

[0039] 2. The present invention is guided by the value function and adopts the preference tree of the best priority search algorithm to automatically construct a collection step-level preference pairing data training set, which does not require manual workload and has great application significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 It is a design flow chart of a large model reasoning path verifier based on a preference tree provided by the present invention;

[0042] Figure 2 It is a preference tree construction flow chart provided by the present invention;

[0043] Figure 3 It is an example of data reasoning in mathematics teaching. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0045] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0046] In order to solve the problem that it is difficult for a binary classification training verifier to obtain detailed supervision signals from its training mode, the present invention proposes a design method for a large model inference path verifier based on a preference tree, which can further improve the accuracy of evaluating candidate paths after learning the relative relationship between different inference paths.

[0047] like Figure 1 As shown, the design method of the large model reasoning path verifier based on the preference tree provided by the present invention includes the following steps:

[0048] 1) Value function design: When solving complex reasoning tasks, large models usually need to rely on multiple reasoning steps to gradually solve the problem. A reasoning step refers to the process by which a large model changes from one logical state to another when solving complex reasoning tasks. Each reasoning step contains a clear logical deduction, and by gradually advancing these steps, the problem is finally completely solved. In order to evaluate the quality of each reasoning step of the large language model, the forward-looking ability of the large language model is used to sample several subsequent reasoning paths starting from the reasoning step to be evaluated, and the quality of the reasoning step is quantified based on the proportion of correct paths in these reasoning paths. The specific implementation details are as follows:

[0049] To evaluate the i-th reasoning step y i , using a large language model from y i Start sampling N subsequent reasoning paths, and the reasoning path set is recorded as N can be set to 8. Reasoning step y i The quality of is quantified by the proportion of correct paths, using the following formula:

[0050]

[0051] in, Represents the reasoning step y i The value of a[P i j ] represents the jth reasoning path P i j The result, j = 1, ..., N, g represents the correct answer, 1 [a [P i j ]=g] means that if the jth reasoning path P i j If the result is correct, it takes 1, otherwise it takes 0.

[0052] 2) Preference tree construction: In order to provide accurate step-level preference feedback, the present invention implements a best-priority tree search algorithm to generate paired data that is crucial for preference learning and construct it as a preference paired data training set. Figure 2 As shown in Figure 1, the construction of the preference tree starts with the original question stem as the root node, where each node represents a reasoning step. Each iteration starts from the root node of the tree and includes the following sub-steps:

[0053] 2.1 Calculate the value of each child node under the current node according to the value function in step 1), and select the node with the highest value for further expansion;

[0054] 2.2 After selecting the node with the highest value, the preference tree is expanded by generating new child nodes. To this end, the large language model is used to sample several potential subsequent reasoning steps, each of which becomes a new child node of the previously selected node with the highest value. If the selected node represents the last step, the generation of an inference path ends, the expansion phase is omitted, and the current iteration ends; the next iteration starts again from the root node of the tree, and all nodes generated in the previous iteration do not participate in the node comparison of this iteration. The specific implementation details are as follows:

[0055] After determining the node with the highest value in step 2.1, we use the large language model π θ Sample k potential subsequent reasoning steps k can be set to 3, x represents the original question, y 1:iRepresents the 1st to the i-th reasoning steps.

[0056] 3) Paired data collection: Create a preference paired data training set containing the expression of reasoning path preferences from the preference tree obtained in step 2), which includes the following sub-steps:

[0057] 3.1 Trace each reasoning path from the root node to the leaf node in the preference tree, compare the value of each child node of each node, and combine these comparison results in pairs to construct a preference pairing data training set; the specific implementation details are as follows:

[0058] In the preference pairing data training set, each entry is in the form of {(x,y + ,y - )}, where x represents the original question stem, y + is the reasoning path as a positive example, and y - is the reasoning path as a negative example;

[0059] At each node in the tree, pairwise comparisons are performed with sibling nodes. Sibling nodes are nodes from the same preceding context y 1:i-1 The possible subsequent reasoning steps branched out. According to the node value obtained in step 2.1, the relative relationship between these reasoning steps can be obtained to form a preference pair, i.e., paired data. For the better step in each preference pair The set of subsequent reasoning paths obtained by sampling in step 1) Select a reasoning path with the correct result to form the reasoning path y + On the contrary, for the less optimal step In its potential subsequent reasoning path set Select a reasoning path with an incorrect result to form a reasoning path y - ;

[0060] 3.2 According to the threshold, filter out the paired data in the training set whose value difference is greater than the threshold; the specific implementation details are as follows: If in step 3.1, a better step With a less optimal step If the value difference between them is greater than the minimum threshold α, then the pairing data {(x,y + ,y - )}, α can be set to 0.375.

[0061] 4) Validator training: The validator is based on a large language model, with a randomly initialized linear layer added, which outputs a scalar value. The preference pairing data training set obtained in step 3) is used to train the validator for preference learning; the specific implementation details are as follows:

[0062] The validator is trained using a step-level ranking loss function that aims to encourage the selection of the most efficient inference path, defined as follows:

[0063]

[0064] Where d is the step subscript where the two reasoning paths diverge, n is the subscript of the last reasoning step, σ(·) represents the sigmoid activation function, and r φ (x,y 1:i ) represents the verifier’s response to the input x and the inference step sequence {y1,…,y i}The output score is given by φ.

[0065] 5) Best reasoning path selection: After using the target large language model to generate several reasoning paths for a question, these candidate paths are used as input to the verifier, and the verifier is asked to score them in turn, and the path with the highest score is selected as the final reasoning path. The specific implementation details are as follows:

[0066] For a given input x, the target large language model generates M candidate reasoning paths {y (1) ,y (2) ,…,y (M)}, M can be set to 64. These reasoning paths are then ranked by the trained verifier, and the one with the highest score is selected as the most credible reasoning path.

[0067] After the verifier training is completed, we can apply the verifier to two reasoning tasks, mathematical reasoning and common sense reasoning, and test its effect on the most commonly used open mathematical reasoning datasets GSM8K and MATH, and common sense reasoning datasets CSQA and StrategyQA, respectively. The specific indicator is the accuracy of the verifier, and the test results are shown in Table 1 and Table 2. Compared with the result supervision verifier and the process supervision verifier, the method of the present invention performs best and is more effective.

[0068] Table 1 Evaluation results of mathematical reasoning tasks

[0069] method GSM8K MATH Results Monitoring 70.74 14.40 Process supervision 74.91 21.20 Method of the present invention 82.79 26.80

[0070] Table 2 Evaluation results of common sense reasoning tasks

[0071] method CSQA StrategyQA Results Monitoring 63.96 67.14 Process supervision 70.11 79.22 Method of the present invention 72.97 83.25

[0072] In the following examples, mathematics teaching data reasoning is taken as an example.

[0073] 1) If Figure 3 As shown in the figure, taking the math problem stem as the root node, the large language model is used to sample three potential subsequent reasoning steps, each of which becomes a new child node of the previously selected highest value node;

[0074] 2) If Figure 3 As shown, the value function is used to evaluate each reasoning step and calculate the value of each node. In this example, the values ​​of the three reasoning steps are 0.875, 0.5, and 0.125 respectively.

[0075] 3) Select the node with the highest value for subsequent sampling until the selected node is the last step in the reasoning path.

[0076] The present invention also provides a design system of a large model reasoning path verifier based on a preference tree, the system comprising the following modules:

[0077] Value function design module: Using a large language model, we sample several subsequent reasoning paths starting from the reasoning step to be evaluated, and quantify the quality of the reasoning step based on the proportion of correct paths in the sampled reasoning paths to obtain the value of the reasoning step;

[0078] Preference tree construction module: Starting from the original question stem as the root node, the preference tree is constructed. Each node represents a reasoning step. Each round of iteration starts from the root node of the tree, including: calculating the value of each child node under the current node, selecting the node with the highest value, and expanding the preference tree by generating new child nodes; using a large language model to sample several potential subsequent reasoning steps, each of which becomes a new child node of the previously selected node with the highest value; if the selected node represents the last step, the generation of a reasoning path ends, and the expansion stage is omitted, and the current round of iteration ends; the next round of iteration starts again from the root node of the tree, and all nodes generated in the previous round of iteration do not participate in the node comparison of this round of iteration;

[0079] Preference pairing data training set construction module: trace each reasoning path from the root node to the leaf node in the preference tree, compare the value of each child node of each node, and combine these comparison results in pairs to construct the preference pairing data training set; filter out the pairing data in the training set whose value difference is greater than the threshold;

[0080] Validator training module: The validator is based on a large language model and adds a linear layer with randomly initialized output scalar values. The validator is trained for preference learning using the preference pairing data training set.

[0081] Best reasoning path selection module: Use the target large language model to generate several reasoning paths for the problem, use these candidate paths as verifier input, let the verifier score them in turn, and select the path with the highest score as the final reasoning path.

[0082] The present invention also provides a design device for a large model reasoning path verifier based on a preference tree, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the above-mentioned design method for the large model reasoning path verifier based on the preference tree.

[0083] The present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned design method of the large model reasoning path verifier based on the preference tree is implemented.

[0084] The above is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the art can make many possible changes and modifications to the technical solution of the present invention by using the above disclosed methods and technical contents without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A design method for a large model reasoning path verifier based on a preference tree, characterized in that: The following steps are involved: (1) Using a large language model, we sample several subsequent reasoning paths starting from the reasoning step to be evaluated, and quantify the quality of the reasoning step based on the proportion of correct paths in the sampled reasoning paths to obtain the value of the reasoning step; (2) Starting from the original question stem as the root node, a preference tree is constructed, where each node represents a reasoning step. Each iteration starts from the root node of the tree and includes the following sub-steps: (2.1) Calculate the value of each child node under the current node and select the node with the highest value for further expansion; (2.2) After selecting the highest value node, expand the preference tree by generating new child nodes; Use the large language model to sample several potential subsequent reasoning steps, and each reasoning step becomes a new child node of the previously selected highest value node; if the selected node represents the last step, the generation of an inference path ends, and the expansion stage is omitted, and the current iteration ends; the next iteration starts again from the root node of the tree, and all nodes generated in the previous iteration do not participate in the node comparison of this iteration; (3) Create a preference pairing data training set containing the preference of the reasoning path from the preference tree obtained in step (2), including the following sub-steps: (3.1) Trace each reasoning path from the root node to the leaf node in the preference tree, compare the value of each child node of each node, and combine these comparison results in pairs to construct a preference pairing data training set; (3.2) Screen out the paired data with a value difference greater than a threshold in the preferred paired data training set; (4) training a verifier, wherein the verifier is based on a large language model and adds a randomly initialized linear layer, which outputs a scalar value; Using the preference pairing data training set obtained in step (3) to perform preference learning training on the verifier; (5) Use the target large language model to generate several reasoning paths for the question, use these candidate paths as input to the verifier, and let the verifier score them in turn, and select the path with the highest score as the final reasoning path.

2. The design method of the large model reasoning path verifier based on the preference tree according to claim 1 is characterized in that: Step (1) is as follows: let the i-th reasoning step be y i , using a large language model from y i Start sampling N subsequent reasoning paths, and the set of sampled reasoning paths is recorded as Reasoning step y i The value of The calculation formula is as follows: Among them, a[P i j ] represents the jth reasoning path P i j The result, j = 1, ..., N, g represents the correct answer, 1 [a [P i j ]=g] means that if the jth reasoning path P i j If the result is correct, it takes 1, otherwise it takes 0.

3. The design method of the large model reasoning path verifier based on preference tree according to claim 2 is characterized in that: In step (1), the potential subsequent reasoning steps for sampling are as follows: after determining the highest value node, the large language model π θ Sample k potential subsequent reasoning steps j=1,…,k, x represents the original question, y 1:i Represents the 1st to the i-th reasoning steps.

4. The design method of a large model reasoning path verifier based on a preference tree according to claim 1 is characterized in that: In step (3.1), the construction of the preference pairing data training set is as follows: Each entry in the preference pairing data training set is in the form of {(x,y + ,y - )}, where x represents the original question stem, y + is the reasoning path as a positive example, y - is the reasoning path as a negative example; At each node in the preference tree, pairwise comparisons are made with sibling nodes; sibling nodes are various possible subsequent reasoning steps branching from the same preceding context; According to the node value obtained in step (2.1), the relative relationship between these reasoning steps is obtained to form a preference pair; for the better step in each preference pair By selecting a correct reasoning path from the set of subsequent reasoning paths sampled in step (1), we can form a reasoning path y + ; For less optimal steps Select a reasoning path with an incorrect result from its potential subsequent reasoning path set to form a reasoning path y - .

5. The design method of the large model reasoning path verifier based on preference tree according to claim 3 is characterized in that: In step (3.2), if and If the value difference between them is greater than the preset threshold α, the pairing data {(x,y + ,y - )}.

6. The design method of the large model reasoning path verifier based on preference tree according to claim 4 is characterized in that: In step (4), the verifier is trained using a step-level ranking loss function, which is designed to promote the selection of the most effective reasoning path. The formula is as follows: Where d is the step subscript where the two reasoning paths diverge, n is the subscript of the last reasoning step, σ(·) represents the sigmoid activation function, and r φ (x,y 1:i ) represents the verifier’s response to the input x and the inference step sequence {y1,…,y i }The output score is given by φ.

7. The design method of a large model reasoning path verifier based on a preference tree according to claim 1, characterized in that: In step (5), for a given input x, the target large language model generates M candidate reasoning paths {y (1) ,y (2) ,…,y (M) }, these reasoning paths are then ranked by the trained verifier, and the one with the highest score is selected as the most credible reasoning path.

8. A design system for a large model reasoning path verifier based on a preference tree, characterized in that: Includes the following modules: Value function design module: Using a large language model, we sample several subsequent reasoning paths starting from the reasoning step to be evaluated, and quantify the quality of the reasoning step based on the proportion of correct paths in the sampled reasoning paths to obtain the value of the reasoning step; Preference tree construction module: Starting from the original question stem as the root node, the preference tree is constructed. Each node represents a reasoning step. Each round of iteration starts from the root node of the tree, including: calculating the value of each child node under the current node, selecting the node with the highest value, and expanding the preference tree by generating new child nodes; using a large language model to sample several potential subsequent reasoning steps, each of which becomes a new child node of the previously selected node with the highest value; if the selected node represents the last step, the generation of a reasoning path ends, and the expansion stage is omitted, and the current round of iteration ends; the next round of iteration starts again from the root node of the tree, and all nodes generated in the previous round of iteration do not participate in the node comparison of this round of iteration; Preference pairing data training set construction module: trace each reasoning path from the root node to the leaf node in the preference tree, compare the value of each child node of each node, and combine these comparison results in pairs to construct the preference pairing data training set; filter out the pairing data in the training set whose value difference is greater than the threshold; Validator training module: The validator is based on a large language model and adds a linear layer with randomly initialized output scalar values. The validator is trained for preference learning using the preference pairing data training set. Best reasoning path selection module: Use the target large language model to generate several reasoning paths for the problem, use these candidate paths as verifier input, let the verifier score them in turn, and select the path with the highest score as the final reasoning path.

9. A design device for a large model reasoning path verifier based on a preference tree, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it is used to implement the design method of a large model reasoning path verifier based on a preference tree as described in any one of claims 1-8.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the method for designing a large model reasoning path verifier based on a preference tree as described in any one of claims 1 to 8 is implemented.

Images