Method for improving reasoning ability of large language model based on Monte Carlo tree search

Abstract

The invention relates to a method for improving the reasoning ability of a large language model based on Monte Carlo tree search. The method comprises the following steps that S1, a mathematical question set and corresponding standard answers are preset; s2, taking each step for solving a mathematical problem as a tree node, and generating a Monte Carlo tree in the whole problem solving process; s3, generating training data based on Monte Carlo tree search; s4, scoring the generated training data; and S5, constructing a loss function according to scores obtained by scoring, optimizing the loss function and updating parameters of the large language model. The reasoning ability of the large language model can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of large language model reasoning, and in particular to a method for improving the reasoning capability of a large language model based on Monte Carlo tree search. Background Art

[0002] The current large language models have far more knowledge than humans, but their logical reasoning ability is far worse than that of humans. They cannot really solve problems like humans by thinking for a long time, such as competition-level math problems. Thinking chain prompts can improve the reasoning ability of large language models to a certain extent, but the improvement is very limited. Methods such as tree search can improve reasoning ability to a greater extent, but the computational cost when used will increase many times.

[0003] Therefore, it is necessary to provide a method for improving the reasoning ability of large language models based on Monte Carlo tree search to improve the reasoning ability of large language models. Summary of the invention

[0004] The purpose of the present invention is to provide a method for improving the reasoning ability of a large language model based on Monte Carlo tree search, so as to improve the reasoning ability of the large language model.

[0005] In order to solve the problems existing in the prior art, the present invention provides a method for improving the reasoning ability of a large language model based on Monte Carlo tree search, comprising the following steps:

[0006] S1: Preset math problem sets and corresponding standard answers;

[0007] S2: Treat each step of solving a mathematical problem as a tree node, and generate a Monte Carlo tree for the entire problem-solving process;

[0008] S3: Generate training data based on Monte Carlo tree search;

[0009] S4: score the generated training data;

[0010] S5: Construct a loss function based on the scores obtained by scoring, optimize the loss function and update the parameters of the large language model.

[0011] Optionally, in the method for improving the reasoning capability of a large language model based on Monte Carlo tree search, generating a Monte Carlo tree includes the following steps:

[0012] S21: The current step is the root node. Starting from the current step, select the child node with the highest confidence upper bound until the current node is the final step or the child nodes of the current node are not full;

[0013] S22: If the current node is not the final step and the child nodes are not full, sample a child node using the large language model;

[0014] S23: Starting from the sampled child node, sample the subsequent problem-solving process until the final answer is obtained;

[0015] S24: Compare the final answer with the standard answer, propagate the reward value from the passed nodes back to the root node, and update the number of visits and cumulative reward values ​​of the passed nodes;

[0016] S25: Repeat S21-S24 to form a Monte Carlo tree.

[0017] Optionally, in the method for improving the reasoning ability of a large language model based on Monte Carlo tree search, the final answer is compared with the standard answer. If the final answer is correct, the reward value is 1; if the final answer is incorrect, the reward value is 0.

[0018] Optionally, in the method for improving the reasoning capability of a large language model based on Monte Carlo tree search, scoring is performed based on the accuracy of the generated training data.

[0019] Optionally, in the method for improving the reasoning capability of a large language model based on Monte Carlo tree search, the loss function is a weighted cross entropy loss function.

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] (1) The present invention can gradually enhance the reasoning ability of a large language model without increasing the computational cost of using the model.

[0022] (2) The output content of the large language model of the present invention is more organized and more similar to the human thinking process.

[0023] (3) The reasoning ability of the present invention can be transferred to other logical reasoning problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of an iterative training model based on Monte Carlo tree search provided by an embodiment of the present invention;

[0025] Figure 2 A flowchart of a method for improving the reasoning capability of a large language model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.

[0027] Hereinafter, if the method described herein includes a series of steps, the order in which the steps are presented herein is not necessarily the only order in which the steps may be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0028] The current large language models have far more knowledge than humans, but their logical reasoning ability is far worse than that of humans. They cannot really solve problems like humans by thinking for a long time, such as competition-level math problems. Thinking chain prompts can improve the reasoning ability of large language models to a certain extent, but the improvement is very limited. Methods such as tree search can improve reasoning ability to a greater extent, but the computational cost when used will increase many times.

[0029] In order to solve the problems existing in the prior art, the present invention provides a method for improving the reasoning ability of a large language model based on Monte Carlo tree search, such as Figure 1 and 2 As shown, the method for improving the reasoning ability of a large language model includes the following steps:

[0030] S1: Preset math problem sets and corresponding standard answers;

[0031] S2: Treat each step of solving a mathematical problem as a tree node, and generate a Monte Carlo tree for the entire problem-solving process;

[0032] S3: Generate training data based on Monte Carlo tree search;

[0033] S4: Score the generated training data. Figure 1 In the Monte Carlo tree of , the larger the number, the higher the score;

[0034] S5: construct a loss function based on the scores obtained by scoring, optimize the loss function and update the parameters of the large language model, wherein the loss function is a weighted cross entropy loss function, in which the weight is set according to the weight of each data.

[0035] The present invention is a method for iteratively training a large language model, such as Figure 1 As shown, in each iteration, the model obtained after the previous iteration is first used to generate data, and then the model is trained on the generated data.

[0036] Continue to refer Figure 1 , in S2, generating a Monte Carlo tree includes the following steps:

[0037] S21: The current step is the root node. Starting from the current step, select the child node with the highest confidence upper bound until the current node is the final step or the child nodes of the current node are not full;

[0038] S22: If the current node is not the final step and the child nodes are not full, sample a child node using the large language model;

[0039] S23: Starting from the sampled child node, sample the subsequent problem-solving process until the final answer is obtained;

[0040] S24: Compare the final answer obtained with the standard answer, propagate the reward value from the passed nodes back to the root node, and update the number of visits and cumulative reward values ​​of the passed nodes; further, compare the final answer obtained with the standard answer, if the final answer is correct, the reward value is 1, if the final answer is wrong, the reward value is 0;

[0041] S25: Repeat S21-S24 to form a Monte Carlo tree.

[0042] Optionally, in the method for improving the reasoning ability of a large language model based on Monte Carlo tree search, scoring is performed based on the accuracy of the generated training data. Specifically, child nodes selected based on the root node are scored, and the scoring basis is the average accuracy of each child node.

[0043] In summary, compared with the prior art, the present invention has the following advantages:

[0044] (1) The present invention can gradually enhance the reasoning ability of a large language model without increasing the computational cost of using the model.

[0045] (2) The output content of the large language model of the present invention is more organized and more similar to the human thinking process.

[0046] (3) The reasoning ability of the present invention can be transferred to other logical reasoning problems.

[0047] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A method for improving the reasoning ability of a large language model based on Monte Carlo tree search, characterized in that: The following steps are involved: S1: Preset math problem sets and corresponding standard answers; S2: Treat each step of solving a mathematical problem as a tree node, and generate a Monte Carlo tree for the entire problem-solving process; S3: Generate training data based on Monte Carlo tree search; S4: score the generated training data; S5: Construct a loss function based on the scores obtained by scoring, optimize the loss function and update the parameters of the large language model.

2. The method for improving the reasoning ability of a large language model based on Monte Carlo tree search according to claim 1, characterized in that: Generating a Monte Carlo tree involves the following steps: S21: The current step is the root node. Starting from the current step, select the child node with the highest confidence upper bound until the current node is the final step or the child nodes of the current node are not full; S22: If the current node is not the final step and the child nodes are not full, sample a child node using the large language model; S23: Starting from the sampled child node, sample the subsequent problem-solving process until the final answer is obtained; S24: Compare the final answer with the standard answer, propagate the reward value from the passed nodes back to the root node, and update the number of visits and cumulative reward values ​​of the passed nodes; S25: Repeat S21-S24 to form a Monte Carlo tree.

3. The method for improving the reasoning ability of a large language model based on Monte Carlo tree search according to claim 2, characterized in that: Compare the final answer with the standard answer. If the final answer is correct, the reward value is 1. If the final answer is wrong, the reward value is 0.

4. The method for improving the reasoning ability of a large language model based on Monte Carlo tree search according to claim 1, characterized in that: Scoring is performed based on the accuracy of the generated training data.

5. The method for improving the reasoning ability of a large language model based on Monte Carlo tree search according to claim 1, characterized in that: The loss function is a weighted cross entropy loss function.

Images