Large code model self-evolution method based on Monte Carlo tree search

Abstract

The invention provides a code large model self-evolution method based on Monte Carlo tree search, which comprises the following steps: S1, inputting a code generation task into a pre-training reasoning model, and generating a reasoning step according to a reasoning task; s2, constructing an inference tree, and obtaining an optimal inference path and a reward score according to an improved Monte Carlo tree search algorithm; s3, according to the optimal reasoning path, based on the automatically generated reasoning and reward data, performing pre-training model self-evolution to improve the reasoning model; and S4, enabling the pre-training model to generate a code of the reasoning scheme, and training the pre-training model to obtain a code large model. According to the method, under the condition that a large amount of fine adjustment data is not needed, the capacity and accuracy of large model code generation can be effectively improved, and the adaptability of the model in a complex reasoning task is further enhanced.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and deep learning technology, and in particular to a code large model self-evolution method based on Monte Carlo tree search. Background Art

[0002] With the development of deep learning and natural language processing technology, code generation tasks have become an important research direction. We define code generation tasks as reasoning tasks plus translation tasks. The reasoning task focuses on how to generate the correct code logic steps based on the given context or problem; while the translation task focuses on code conversion between natural language and programming language or between different programming languages.

[0003] Traditional code generation methods often rely on large-scale pre-trained language models. These models are trained on a large amount of code data and can generate relatively accurate code [1]. However, these methods often rely on the initial performance of the model and lack a self-improvement mechanism during the reasoning process. That is, when the model performs poorly on complex code tasks, it is usually necessary to manually adjust or retrain the model, resulting in low efficiency and increased development costs [2].

[0004] The existing code generation methods still have the following shortcomings:

[0005] (1) Lack of an effective reasoning step generation mechanism: Traditional code generation methods are unable to dynamically generate and optimize reasoning steps, resulting in poor performance in complex task scenarios.

[0006] (2) Insufficient model self-evolution capability: Existing methods usually rely on fixed pre-trained models and lack self-learning and improvement mechanisms in reasoning tasks.

[0007] (3) Low data utilization: In the reasoning step of code generation, existing methods fail to fully utilize the generated reasoning steps and reward signals to further optimize the model.

[0008] [1]Vaswani, A., et al. "Attention Is All You Need." Advances in Neural Information Processing Systems, 2017.

[0009] [2]Radford, A., et al. "Language Models are Few-Shot Learners." arXivpreprint arXiv:2005.14165, 2020. Summary of the invention

[0010] In order to solve the above problems, without increasing the number of model parameters or changing the model structure, the present invention improves the Monte Carlo Tree Search (MCTS), a search algorithm widely used in optimization problems, hereinafter referred to as AutoMCTS, and then combines AutoMCTS with a large-scale pre-trained model to gradually generate and optimize the reasoning steps, and further trains the model through the generated reward signal, thereby realizing self-evolution in code generation tasks.

[0011] The code large model self-evolution method based on Monte Carlo tree search proposed by the present invention comprises:

[0012] S1: Input the code generation task into the reasoning model in the form of reasoning tasks , generate reasoning steps according to the reasoning task;

[0013] S2, construct an inference tree and obtain the optimal inference path and reward score according to the improved Monte Carlo tree search algorithm;

[0014] S3: Based on the optimal reasoning path, the reasoning model is based on the automatically generated reasoning and reward data self-evolution;

[0015] S4. Based on the (question, reasoning solution) data pairs generated by the improved Monte Carlo tree search algorithm, let the pre-trained model Generate the code for the inference solution, convert the (question, inference solution) data pair into the (question, inference solution + code) data pair, and use the (question, inference solution + code) data pair to train the pre-trained model , get the code model .

[0016] Furthermore, in step S1, the pre-trained model is used As an inference model , generating reasoning steps.

[0017] Furthermore, in step S2, the method for constructing the inference tree includes:

[0018] The reasoning task is represented as a tree, in which each node represents a reasoning step, and each edge represents the reasoning process from the previous reasoning step to the next reasoning step. The root node is the description of the problem, and the leaf node is the final solution.

[0019] Furthermore, in step S2, an improved Monte Carlo tree search algorithm is used to search for the optimal path in the inference tree, and the improvement is that:

[0020] For each expanded node, use the inference model For problem-step pairs, generate reflections on step n+1, which are used to guide the specific generation of step n+1 in the subsequent process to ease the reasoning model Weakness to long-context thinking.

[0021] Furthermore, in step S2, an improved Monte Carlo tree search algorithm is used to search for the optimal path in the inference tree, and the improvement is:

[0022] Reward scoring is performed for Step n; scoring principles include: single-step correctness, continuity, and overall effectiveness of the solution.

[0023] Furthermore, in step S2, an improved Monte Carlo tree search algorithm is used to search for the optimal path in the inference tree. The improvement is that: The scoring principle is as follows: if there are errors in the reasoning steps, a relatively low score will be obtained; if there are no errors in the reasoning steps, but there are errors in the previous historical steps, which may cause the subsequent steps to go in the wrong direction, then no high score will be obtained; if the reasoning steps are correct, and are logically continuous with the previous steps, and can solve the problem in the next one or more generations, then a higher score should be obtained; only when the current solution can completely solve the problem can a high score be obtained.

[0024] Furthermore, in step S2, a quality score is assigned to each solution, and the process is as follows:

[0025] (1) Logic verification: The model verifies the logical correctness of a single reasoning step and the continuity of the previous and subsequent steps. If the output does not meet expectations, the model records the low score of the path so that its priority can be reduced in subsequent MCTS iterations.

[0026] (2) Performance evaluation: Evaluate the performance of the solution through time complexity analysis; if the generated solution is inefficient, the inference model It will try to generate more efficient solutions through different reasoning paths;

[0027] (3) Quality scoring: Based on the above evaluation, the inference model A quality score is assigned to each solution to reflect the overall quality of the solution.

[0028] Furthermore, after the initial solution is generated and evaluated, the improved Monte Carlo Tree Search itself iterates through a feedback mechanism.

[0029] Further, in step S3, the reasoning path generated by the Monte Carlo tree search algorithm is input into the reasoning model , so that it evolves itself, the method is as follows:

[0030] Take the inference path as input and train the inference model , in order to learn to generate logically rigorous and clearly organized reasoning steps, so that this path can be given priority in subsequent reasoning tasks.

[0031] Furthermore, it also includes:

[0032] Before executing step S4, repeat steps S2 and S3 to iterate and obtain the final inference model. .

[0033] The model self-evolution method based on Monte Carlo tree search proposed in the present invention has the following beneficial effects:

[0034] (1) Enhanced reasoning capability: Through the search optimization of AutoMCTS, the generated reasoning steps are more in line with the target, significantly improving the accuracy of code generation.

[0035] (2) Model self-evolution: By repeatedly training the inference model and reward model, the system can continuously improve its performance and adapt to more complex reasoning tasks.

[0036] (3) High degree of automation: The method of the present invention reduces the need for human intervention and improves the level of automation in the code generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A schematic diagram of a process of a self-evolution method according to an embodiment of the present invention;

[0039] Figure 2 A schematic diagram showing the comparison between MCTS and AutoMCTS according to an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of a reasoning task test flow according to an embodiment of the present invention;

[0041] Figure 4 FIG. 4 is a flow chart of the evaluation phase in AutoMCTS according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be checked and described in full in combination with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The present invention proposes a code large model self-evolution method based on Monte Carlo tree search, such as Figure 1 As shown, including:

[0044] S1: Input the code generation task into the reasoning model in the form of reasoning tasks , generate reasoning steps according to the reasoning task;

[0045] S2, construct an inference tree and obtain the optimal inference path and reward score according to the improved Monte Carlo tree search algorithm;

[0046] S3: Based on the optimal reasoning path, the model is self-evolved based on the automatically generated reasoning and reward data to improve the reasoning model ;

[0047] S4. Based on the (question, reasoning solution) data pairs generated by the improved Monte Carlo tree search algorithm, let the pre-trained model Generate the code for the inference solution, convert the (question, inference solution) data pair into the (question, inference solution + code) data pair, and use the (question, inference solution + code) data pair to train the pre-trained model , get the final code model .

[0048] In step S1, the code generation task is usually driven by a specific problem. For example, suppose we have a code generation task: "Write a function to calculate the factorial of a given number." This task will be input to the inference model In the code generation process, the model will generate a set of inference tasks. If the code generation task is relatively simple, it can be used as an inference task; if the code generation task is relatively complex, it can generate a series of inference tasks.

[0049] Each reasoning task can be decomposed into several reasoning steps. For example, for the factorial problem, the reasoning task can generate the following reasoning steps in the pre-trained model:

[0050] (1) Determine the function name and parameters: factorial(n):

[0051] (2) Initialize the result variable result = 1

[0052] (3) Use a loop to calculate factorial for i in range(1, n + 1):

[0053] (4) Update the result result *= i in the loop

[0054] (5) Return result

[0055] Assume that the input of the reasoning task is X and the output is Y, where X is the question or context description and Y is the natural language reasoning step that is expected to be generated. Note that Y does not include code, and the code in the example is only for illustration. In one embodiment, the pre-trained model As an initial inference model , generates inference steps (no actual code generation) As an intermediate process of reasoning.

[0056] In step S2, the method for constructing the inference tree includes:

[0057] Represent the reasoning task as a tree. In this tree, each node represents a reasoning step, and each edge represents the reasoning process from the previous reasoning step to the next reasoning step. The root node is the initial state (the description of the problem), and the leaf node is the final solution.

[0058] For example, Figure 3 As shown in the figure, for the problem of determining prime factors, the root node is the string describing the task "determine whether an integer is a prime factor", and the leaf nodes are the final solution steps generated.

[0059] In the reasoning task, how to select the optimal reasoning path is a key issue. To solve this problem, the present invention uses the AutoMCTS algorithm to search for the optimal path in the reasoning tree and gradually optimize the reasoning process, such as Figure 2 The present invention innovates on the original MCTS and solves the problem that the original MCTS can only get reward scores after multiple simulations. Specifically, the present invention includes four main steps: selection, expansion, reflection, evaluation and backtracking.

[0060] (1) Selection: Starting from the root node, child nodes are selected according to a strategy until a node that is not fully expanded is reached. The selection strategy usually uses the upper confidence limit formula (UCB):

[0061]

[0062] in, is the sum of the reward values ​​of node i, is the number of times node i is visited, is the number of times the parent node of a node has been visited, and c is a constant that controls the balance between exploration and exploitation.

[0063] (2) Expansion: Select a node that is not fully expanded and generate all its possible child nodes. For reasoning tasks, the child nodes represent different reasoning steps or code snippets.

[0064] (3) Reflection: For each expanded node, use the inference model For question-step pairs (Question, Step 1, ..., Step n), generate reflections on Step n+1, which will be used to guide the generation of step n+1, so as to alleviate the model's weakness in thinking about long contexts. Question is the question itself, and Step 1, ...Step n are all nodes on the path, representing the previous steps. Reflection is a brief description of the next step Sn+1 generated by the pre-trained model for the previous steps S1 to Sn, because if the large model is directly asked to generate Sn+1 for the previous steps S1 to Sn, the generation may be unsatisfactory because the context is too long. When in use, the reflection content is attached to the question and S1 to Sn, and Sn+1 is generated as the context. Adding reflection to MCTS is one of the improvements of the Monte Carlo tree search algorithm of the present invention.

[0065] (4) Evaluation: The evaluation and reflection processes can be performed simultaneously. They both process the newly generated nodes that are expanded, but the evaluation is only for Step n, and the reward score for Step n is calculated. Figure 4 As shown, the present invention innovatively sets several scoring principles, such as: single-step correctness, coherence, and overall effectiveness of the solution. In one embodiment, scoring can be done manually or by training a reasoning model to score based on these principles. For example, the reasoning model can be used directly , set these principles in the interactive prompt, and give 1-2 examples to let the large model learn the input and output format. The principles are as follows: If the new step contains an error, it will receive a relatively low score (such as 1-3 points); There are no errors in the previous steps, but there are errors in the previous steps, which may lead to the wrong direction in the subsequent steps. In this case, you cannot get a high score and should get 3-5 points. If the solution is correct, logically connected to the previous steps, and can solve the problem in the next one or more generations, it should get a score of 5 or more and 7 or less. Only when the current solution can completely solve the problem can it get a score of 8 or more (out of 10).

[0066] (5) Backpropagation: The reward value obtained during the simulation is passed from the leaf node to the root node, and the number of visits and the total reward of each node are updated.

[0067] Through the above process, AutoMCTS can effectively explore multiple possibilities of reasoning steps and find the reasoning path with the highest reward value.

[0068] It is a common method to use a large model as a decision model, but in this invention, we innovatively let the large model play two roles, as a decision maker and a scorer. The scorer role often uses a larger model, but we believe that small models themselves have rich potential, so we use a self-evolution method to stimulate the model's active thinking ability. In this invention, we rely on the inference model The reward score is automatically generated without designing a specific reward function, avoiding artificially compromising robustness. The inference model is not only used to generate inference steps, but also to evaluate the pros and cons of these steps, thus simplifying the entire process and avoiding additional reward model training overhead.

[0069] At each inference step generated by AutoMCTS, the inference model The next possible reasoning step will be generated based on the current context. Assume that the current reasoning step is , the reasoning model is based on context Generate new reasoning steps (including previously generated reasoning steps and the input initial question) .

[0070] Generate a complete reasoning step based on the best reasoning path (the reasoning path with the highest reward value). That is, after the best reasoning path is determined, it is gradually combined in the order of the reasoning path to finally form a complete reasoning step plan.

[0071] After the initial inference solution is generated and evaluated, AutoMCTS itself iterates through a feedback mechanism.

[0072] If some reasoning paths generate low scores, the reasoning model will readjust the reasoning path based on this feedback information. AutoMCTS will continue searching based on the updated reward scores until a better reasoning generation strategy is found. This iterative process is adaptive, allowing the reasoning model to continuously improve the quality and efficiency of code generation.

[0073] In step S3, a large number of inference paths generated by the AutoMCTS algorithm are used to input these data into the inference model , so that it can evolve itself, that is, continuously improve its reasoning ability through retraining, as follows:

[0074] After multiple reasoning tasks of the Monte Carlo Tree Search algorithm, a wealth of reasoning data will be accumulated. This data includes multiple reasoning paths for different problems and the performance of each path under different conditions. Reasoning data usually includes the following parts:

[0075] Problem Description: A description of the original problem, such as "Write a function to calculate factorial".

[0076] Reasoning Path: The sequence of reasoning steps generated by MCTS.

[0077] Reward score: The cumulative reward score corresponding to each reasoning path.

[0078] After the inference data has accumulated to a certain extent, the inference model Retraining is to make the model A better reasoning path can be generated in subsequent tasks.

[0079] The retraining process usually adopts supervised learning, taking the inference path as input and training the inference model. The model will learn which reasoning steps can generate high-scoring codes, and thus prioritize these paths in subsequent reasoning tasks. Through retraining, the model's reasoning ability will gradually improve, allowing the model to more accurately select reasoning paths and generate higher-quality solutions, thereby achieving the effect of self-evolution.

[0080] Inference Model Only used in the evolutionary process in AutoMCTS to generate data.

[0081] In one embodiment, S2 and S3 are repeatedly executed to iterate and obtain the final inference model. . Each iteration, the inference model The above is the subsequent iteration process starting from the second iteration. The overall process of the first iteration remains unchanged, but the inference model Initialized as a pre-trained model .

[0082] In step S4, AutoMCTS generates a lot of (question, reasoning solution) data pairs, which are expanded to (question, reasoning solution, code) data triplets in this step. After conversion, they are used to train the pre-trained model called the code large model. The specific steps are as follows:

[0083] Input (question, reasoning solution) data pairs into the pre-trained model ,Model Generate the corresponding code according to the reasoning solution. Then transform the (question, reasoning solution, code) data triple into (question, reasoning solution + code) data pair as training data. ,get .

[0084] Embodiment 1

[0085] Experiments were conducted on the evaluation indicators Human-Eval and MBPP datasets. In order to evaluate the effect of this technology, evaluation experiments were conducted on the above datasets.

[0086] The experimental results are shown in the figure below:

[0087] Model average Human-Eval MBPP Qwen2-7B-Instruct 65.52 77.44 53.60 Mistral-7B-V0.3-Instruct 39.7 37.80 41.60 Meta-Llama-3.1-8B-Instruct 62.02 64.63 59.40 Our Model <![CDATA[ 66.84 ]]> <![CDATA[ 79.27 ]]> 54.40

[0088] The data in the table are the pass@1 values ​​of different models on various data sets. The higher the value, the better the model effect. The bold data are the best models. The underlined data indicate that our method exceeds the traditional method in this indicator.

[0089] Experimental results show that our model outperforms traditional models in various tasks, which fully demonstrates the effectiveness of the model self-evolution method based on Monte Carlo tree search. At the same time, our method has lower requirements on the quantity and quality of data.

[0090] The present invention fully stimulates the reasoning ability of the large model through our improved AutoMCTS method, allowing the large model to improve autonomously, and finally obtains a large code model that can solve complex problems. It outperforms benchmark models such as Qwen2-7B-Instruct in multiple downstream tasks, and does not add additional parameters. It does not require a large amount of high-quality fine-tuning data, avoids massive overhead in data acquisition, and is lower in cost and better in effect than existing methods, thus making a certain contribution to the development of large code models.

[0091] Functional features: The final product of training, the code model, can be used as a new large model in the code field. At the same time, the reasoning data and code data obtained from training can also provide support for other tasks.

[0092] Those skilled in the art will appreciate that the foregoing embodiments are merely intended to illustrate the technical solutions of the present invention rather than to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that the technical solutions described in the foregoing embodiments may be modified or some or all of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope defined by the claims of the present invention.

Claims

1. A code large model self-evolution method based on Monte Carlo tree search, characterized in that: The generation method comprises: S1: Input the code generation task into the reasoning model in the form of reasoning tasks, and generate reasoning steps according to the reasoning tasks; S2, construct an inference tree and obtain the optimal inference path and reward score according to the improved Monte Carlo tree search algorithm; S3. According to the optimal reasoning path, the reasoning model is self-evolved based on the automatically generated reasoning and reward data; S4. Based on the (problem, reasoning solution) data pairs generated by the improved Monte Carlo tree search algorithm, let the pre-trained model generate the code of the reasoning solution, convert the (problem, reasoning solution) data pairs into (problem, reasoning solution + code) data pairs, use the (problem, reasoning solution + code) data pairs to train the pre-trained model, and obtain the large code model.

2. The code big model self-evolution method according to claim 1, characterized in that: In step S1, a pre-trained model is used as an inference model to generate inference steps.

3. The code big model self-evolution method according to claim 1, characterized in that: In step S2, the method for constructing the inference tree includes: The reasoning task is represented as a tree, in which each node represents a reasoning step, and each edge represents the reasoning process from the previous reasoning step to the next reasoning step. The root node is the description of the problem, and the leaf node is the final solution.

4. The code big model self-evolution method according to claim 1, characterized in that: In step S2, an improved Monte Carlo tree search algorithm is used to search for the optimal path in the inference tree. The improvement is: For each expanded node, the reasoning model is used to generate reflections on the n+1th step for the problem-several step pairs, which are used to guide the specific generation of the n+1th step in the subsequent process to alleviate the weakness of the reasoning model for long-context thinking.

5. The code big model self-evolution method according to claim 1, characterized in that: In step S2, an improved Monte Carlo tree search algorithm is used to search for the optimal path in the inference tree. The improvement is: Reward scoring is performed for Step n; scoring principles include: single-step correctness, continuity, and overall effectiveness of the solution.

6. The method for self-evolution of a large code model according to claim 5, characterized in that: In step S2, an improved Monte Carlo tree search algorithm is used to search for the optimal path in the inference tree. The improvement lies in: scoring the inference model according to the following principles: if there is an error in the inference step, a relatively low score will be obtained; if there is no error in the inference step, but there is an error in the previous historical step, which may lead to the subsequent wrong direction, then a high score cannot be obtained; if the inference step is correct, and is logically continuous with the previous step, and can solve the problem in the next one or more generations, then a higher score should be obtained; Only when the current solution can completely solve the problem can you get a high score.

7. The code big model self-evolution method according to claim 5, characterized in that: In step S2, a quality score is assigned to each solution segment as follows: (1) Logic verification: The model verifies the logical correctness of a single reasoning step and the continuity of the previous and subsequent steps. If the output does not meet expectations, the model records the low score of the path so that its priority can be reduced in subsequent MCTS iterations. (2) Performance evaluation: The performance of the solution is evaluated through time complexity analysis. If the generated solution is inefficient, the reasoning model will try to generate a more efficient solution through different reasoning paths. (3) Quality scoring: Based on the above evaluation, the inference model assigns a quality score to each solution to reflect the overall quality of the solution.

8. The code big model self-evolution method according to claim 5, characterized in that: After an initial solution is generated and evaluated, the Monte Carlo Tree Search itself iterates through a feedback mechanism.

9. The code big model self-evolution method according to claim 1, characterized in that: In step S3, the reasoning path generated by the Monte Carlo tree search algorithm is input into the reasoning model to make it self-evolve, as follows: The reasoning path is taken as input and the reasoning model is trained to learn to generate logically rigorous and clearly organized reasoning steps, so that this path can be given priority in subsequent reasoning tasks.

10. The method for self-evolution of a large code model according to claim 1, characterized in that: Also includes: Before executing step S4, steps S2 and S3 are repeatedly executed to iterate and obtain the final inference model.

Images