A code large model self-evolution method based on monte carlo tree search

Abstract

The application provides a code large model self-evolution method based on Monte Carlo tree search, comprising the following steps: S1, inputting a code generation task into a pre-training inference model, and generating an inference step according to the inference task; S2, constructing an inference tree, and obtaining an optimal inference path and a reward score according to an improved Monte Carlo tree search algorithm; S3, performing self-evolution of the pre-training model based on automatically generated inference and reward data according to the optimal inference path, and improving the inference model; and S4, enabling the pre-training model to generate code of an inference scheme, training the pre-training model, and obtaining a code large model. The application can effectively improve the code generation capability and accuracy of the large model without a large amount of fine-tuning data, and also enhances the self-adaptability of the model in a complex inference task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing and deep learning, and particularly relates to a code large model self-evolution method based on Monte Carlo tree search. BACKGROUND

[0002] With the development of deep learning and natural language processing technology, code generation task has become an important research direction. We define the code generation task as reasoning task plus translation task. Among them, the reasoning task focuses on how to generate correct code logic steps according to the given context or question; and the translation task focuses on code conversion between natural language and programming language or conversion between different programming languages.

[0003] In traditional code generation methods, large-scale pre-training language models are often relied on. These models are trained on a large amount of code data and can generate relatively accurate code [1]. However, these methods often rely on the initial performance of the model, lack of self-improvement mechanism in the reasoning process, that is, when the model performs poorly on complex code tasks, it usually needs to be adjusted or retrained manually, resulting in low efficiency and increased development cost [2].

[0004] The existing code generation method also has the following shortcomings:

[0005] (1) Lack of effective reasoning step generation mechanism: traditional code generation methods cannot dynamically generate and optimize reasoning steps, resulting in poor performance in complex task scenarios.

[0006] (2) Insufficient model self-evolution ability: existing methods usually rely on fixed pre-training models, lack of self-learning and improvement mechanism in reasoning tasks.

[0007] (3) Low data utilization rate: in the reasoning step of code generation, existing methods fail to fully utilize the generated reasoning steps and reward signals to further optimize the model.

[0008] [1] Vaswani, A., et al. "Attention Is All You Need." Advances in Neural Information Processing Systems, 2017.

[0009] [2] Radford, A., et al. "Language Models are Few-Shot Learners." arXiv preprint arXiv:2005.14165, 2020. SUMMARY

[0010] In order to solve the above problems, without increasing the parameter amount of the model and without changing the model structure, the application improves the search algorithm of Monte Carlo Tree Search (MCTS) which is widely used in optimization problems, hereinafter referred to as AutoMCTS, and then combines AutoMCTS with a large-scale pre-training model, gradually generates and optimizes the reasoning steps, and further trains the model through the generated reward signal, to realize self-evolution in the code generation task.

[0011] The code large model self-evolution method based on Monte Carlo Tree Search proposed by the application comprises:

[0012] S1: inputting the code generation task in the form of a reasoning task into a reasoning model , generating a reasoning step according to the reasoning task;

[0013] S2, constructing a reasoning tree, and obtaining an optimal reasoning path and a reward score according to the improved Monte Carlo Tree Search algorithm;

[0014] S3, according to the optimal reasoning path, performing self-evolution of the reasoning model based on automatically generated reasoning and reward data;

[0015] S4, generating a (question, reasoning scheme) data pair according to the improved Monte Carlo Tree Search algorithm, letting the pre-training model generate the code of the reasoning scheme, converting the (question, reasoning scheme) data pair into a (question, reasoning scheme + code) data pair, training the pre-training model using the (question, reasoning scheme + code) data pair, and obtaining a code large model .

[0016] Further, in step S1, a pre-training model is used as the reasoning model to generate the reasoning step.

[0017] Further, in step S2, the construction method of the reasoning tree comprises:

[0018] The reasoning task is represented as a tree, in which each node represents a reasoning step, and each edge represents the reasoning process from the previous reasoning step to the next reasoning step, the root node is the description of the problem, and the leaf node is the final solution.

[0019] Further, in step S2, the improved Monte Carlo Tree Search algorithm is used to search for the optimal path in the reasoning tree, and the improvement lies in:

[0020] For each expanded node, use the inference model For the question-some step pairs, generate reflection on the nth+1 step, which is used in the subsequent process to guide the specific generation of the nth+1 step to alleviate the inference model Weakness for long context thinking.

[0021] Further, in step S2, an improved Monte Carlo tree search algorithm is used to search for the optimal path in the inference tree, and the improvement is that:

[0022] Reward score for Step n; the scoring principles include: single-step correctness, coherence, and overall effectiveness of the scheme.

[0023] Further, in step S2, an improved Monte Carlo tree search algorithm is used to search for the optimal path in the inference tree, and the improvement is that: Score, principles as follows: if the inference step contains errors, it will get a relatively low score; if there is no error in the inference step, but there is an error in the historical previous step, which may lead to subsequent errors, it cannot get a high score; if the inference step is correct and logically continuous with the previous step, and can solve the problem in one or more subsequent generations, it should get a higher score; only when the current scheme can completely solve the problem, can it get a high score.

[0024] Further, in step S2, a quality score is assigned to each segment of the scheme, and the process is as follows:

[0025] (1) Logical verification: the model will verify the logical correctness of a single inference step and the continuity of the previous and subsequent steps; if the output does not meet expectations, the model will record a low score for that path to reduce its priority in subsequent MCTS iterations;

[0026] (2) Performance evaluation: evaluate the performance of the scheme through time complexity analysis; if the generated scheme is relatively inefficient, the inference model will try to generate a more efficient scheme through different inference paths;

[0027] (3) Quality scoring: based on the above evaluation, the inference model assigns a quality score to each segment of the scheme to reflect the overall pros and cons of the scheme.

[0028] Further, after generating the solution and evaluating it for the first time, the improved Monte Carlo tree search itself will iterate through the feedback mechanism.

[0029] Further, in step S3, the inference path generated by the Monte Carlo tree search algorithm is input into the inference model to evolve itself, and the method is as follows:

[0030] Training the reasoning model with the reasoning path as input to learn a reasoning step that is logically sound and well-organized, thereby prioritizing this path in subsequent reasoning tasks.

[0031] Further, it also includes:

[0032] Before performing step S4, steps S2 and S3 are repeatedly performed for iteration to obtain a final reasoning model .

[0033] The model self-evolution method based on Monte Carlo tree search proposed by the present application has the following beneficial effects:

[0034] (1) Enhanced reasoning ability: through the search optimization of AutoMCTS, the generated reasoning steps are more in line with the target, significantly improving the accuracy of code generation.

[0035] (2) Model self-evolution: through repeated training of the reasoning model and the reward model, the system can continuously improve its performance and adapt to more complex reasoning tasks.

[0036] (3) High degree of automation: the method of the present application reduces the need for human intervention and improves the automation level of the code generation process. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0038] Figure 1 It is a flowchart of the self-evolution method according to one embodiment of the present application;

[0039] Figure 2 It is a comparison diagram of MCTS and AutoMCTS according to one embodiment of the present application;

[0040] Figure 3 It is a reasoning task test flowchart according to one embodiment of the present application;

[0041] Figure 4 It is an evaluation stage flowchart in AutoMCTS according to one embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0043] The present application provides a code large model self-evolution method based on Monte Carlo tree search, as shown in Figure 1 , comprising:

[0044] S1: input the code generation task in the form of inference task to the inference model , generate inference steps according to the inference task;

[0045] S2, construct an inference tree, and obtain the optimal inference path and reward score according to the improved Monte Carlo tree search algorithm;

[0046] S3, according to the optimal inference path, based on the automatically generated inference and reward data, model self-evolution is carried out to improve the inference model ;

[0047] S4, according to the (question, inference scheme) data pair generated by the improved Monte Carlo tree search algorithm, let the pre-training model generate the code of the inference scheme, convert the (question, inference scheme) data pair into (question, inference scheme + code) data pair, and use the (question, inference scheme + code) data pair to train the pre-training model , get the final code large model .

[0048] In step S1, the code generation task is usually driven by a specific problem. For example, suppose we have a code generation task: "Write a function to calculate the factorial of a given number." This task will be input to the inference model , the model will generate a set of inference tasks. If the code generation task is relatively simple, it can be used as an inference task; if the code generation task is relatively complex, a series of inference tasks can be generated.

[0049] Each inference task can be decomposed into several inference steps. For example, for the factorial problem, the inference task in the pre-training model can generate the following inference steps:

[0050] (1) Determine the name and parameters of the function: factorial(n):

[0051] (2) Initialize the result variable result = 1

[0052] (3) Use a loop to calculate factorial for i in range(1, n + 1):

[0053] (4) Update the result result *= i in the loop

[0054] (5) Return result

[0055] Assume that the input of the reasoning task is X and the output is Y, where X is the question or context description and Y is the natural language reasoning step expected to be generated. Note that Y does not include code, and the code in the example is only for illustration. In one embodiment, the pre-trained model As an initial inference model , generates inference steps (no actual code generation) As an intermediate process of reasoning.

[0056] In step S2, the method for constructing the inference tree includes:

[0057] Represent the reasoning task as a tree. In this tree, each node represents a reasoning step, and each edge represents the reasoning process from the previous reasoning step to the next. The root node is the initial state (the description of the problem), and the leaf nodes are the final solution.

[0058] For example, Figure 3 As shown in the figure, for the prime factor problem, the root node is the string describing the task "determine whether an integer is a prime factor", and the leaf nodes are the final solution steps.

[0059] In reasoning tasks, how to choose the optimal reasoning path is a key issue. To solve this problem, the present invention uses the AutoMCTS algorithm to search for the optimal path in the reasoning tree and gradually optimize the reasoning process, such as Figure 2 This invention innovates on the original MCTS and solves the problem that the original MCTS can only obtain reward scores after multiple simulations. Specifically, it includes four main steps: selection, expansion, reflection, evaluation, and backtracking.

[0060] (1) Selection: Starting from the root node, child nodes are selected according to a strategy until a node that is not fully expanded is reached. The selection strategy usually uses the upper confidence bound (UCB) formula:

[0061]

[0062] in, is the sum of the reward values ​​of node i, is the number of times node i is visited, is the number of times the parent node of a node has been visited, and c is a constant that controls the balance between exploration and exploitation.

[0063] (2) Expansion: Select a node that is not fully expanded and generate all its possible child nodes. For reasoning tasks, child nodes represent different reasoning steps or code snippets.

[0064] (3) Reflection: For each expanded node, use the inference model For question-step pairs (Question, Step 1, ..., Step n), a reflection on Step n+1 is generated, which will be used to guide the generation of Step n+1, thereby alleviating the model's weakness in thinking about long contexts. Question is the question itself, and Step 1, ...Step n are all nodes on the path, representing the previous steps. Reflection is a brief description of the next step Sn+1 generated by the pre-trained model for the previous steps S1 to Sn, because if the large model is directly asked to generate Sn+1 for the previous steps S1 to Sn, the generation may be suboptimal due to the length of the previous context. When in use, the reflection content is appended to the question and S1 to Sn, and used as the previous context to generate Sn+1. Adding reflection to MCTS is one of the improvements of the present invention to the Monte Carlo tree search algorithm.

[0065] (4) Evaluation: The evaluation and reflection processes can be carried out simultaneously. They both process the newly generated nodes that are expanded, but the evaluation is only for Step n, and the reward score for Step n is given. Figure 4 As shown, the present invention innovatively sets several scoring principles, such as: single-step correctness, consistency, and overall effectiveness of the solution. In one embodiment, scoring can be done manually or by training a reasoning model to score based on these principles. For example, the reasoning model can be used directly , set these principles in the interactive prompt and give 1-2 examples to let the large model learn the input and output format. The principles are as follows: If the inference model is newly generated If there are errors in the new step, a relatively low score (such as 1-3 points) will be obtained. There are no errors in the previous steps, but there are errors in the previous steps, which may lead to the wrong direction in the future. In this case, you cannot get a high score and should score within 3-5 points. If the solution is correct, logically connected to the previous steps, and can solve the problem in the next one or more generations, it should receive a score of 5 or more and 7 or less. Only when the current solution can completely solve the problem can it receive a score of 8 or more (out of 10 points).

[0066] (5) Backpropagation: The reward value obtained during the simulation is passed from the leaf node to the root node, and the number of visits and the total reward of each node are updated.

[0067] Through the above process, AutoMCTS can effectively explore multiple possibilities of reasoning steps and find the reasoning path with the highest reward value.

[0068] Using a large model as a decision model is a common approach, but in this invention, we innovatively let the large model play two roles, acting as both a decision maker and a scorer. The scorer role often uses a larger model, but we believe that small models themselves have rich potential, so we use a self-evolution approach to stimulate the model's proactive thinking ability. In this invention, we rely on the inference model To automatically generate reward scores without designing a specific reward function, this approach avoids artificially compromising robustness. The inference model is used not only to generate inference steps but also to evaluate the quality of these steps, simplifying the entire process and avoiding the additional overhead of reward model training.

[0069] At each inference step generated by AutoMCTS, the inference model The next possible reasoning step will be generated based on the current context. Assume that the current reasoning step is , the reasoning model is based on context Generate new reasoning steps (including the previously generated reasoning steps and the input initial question) .

[0070] A complete set of reasoning steps is generated based on the optimal reasoning path (the one with the highest reward value). In other words, after the optimal reasoning path is determined, the reasoning paths are gradually combined in the order of the reasoning paths to form a complete reasoning step plan.

[0071] After initially generating and evaluating an inference solution, AutoMCTS itself iterates through a feedback mechanism.

[0072] If certain reasoning paths generate low scores, the reasoning model readjusts the reasoning path based on this feedback. AutoMCTS continues searching based on the updated reward scores until it finds a more optimal reasoning generation strategy. This iterative process is adaptive, allowing the reasoning model to continuously improve the quality and efficiency of code generation.

[0073] In step S3, the large number of reasoning paths generated by the AutoMCTS algorithm are input into the reasoning model to evolve itself, i.e., continuously improve its reasoning ability through retraining, as follows:

[0074] After multiple reasoning tasks using the Monte Carlo Tree Search algorithm, a wealth of reasoning data is accumulated. These data include various reasoning paths for different problems and the performance of each path under different conditions. The reasoning data typically includes the following parts:

[0075] • Problem description: Description of the original problem, such as "Write a function to calculate factorial".

[0076] • Reasoning path: Reasoning step sequence generated by MCTS.

[0077] • Reward score: Cumulative reward score corresponding to each reasoning path.

[0078] After the reasoning data accumulates to a certain extent, the reasoning model is retrained, aiming to enable the model to generate higher-quality reasoning paths in subsequent tasks.

[0079] The retraining process typically uses supervised learning, taking reasoning paths as input to train the reasoning model . The model will learn which reasoning steps can generate high-score code, so as to preferentially select these paths in subsequent reasoning tasks. Through retraining, the reasoning ability of the model will gradually improve, enabling the model to more accurately select reasoning paths and generate higher-quality solutions, thereby achieving self-evolution.

[0080] The reasoning model is only used in the evolution process in AutoMCTS to generate data.

[0081] In an embodiment, steps S2 and S3 are repeatedly executed for iteration to obtain the final reasoning model . In each iteration, the reasoning model undergoes retraining. The above introduction describes the subsequent iteration process starting from the second time. The overall process of the first iteration remains unchanged, but the reasoning model is initialized as the pre-trained model .

[0082] In step S4, AutoMCTS generates a large number of (question, reasoning solution) data pairs, which are expanded to (question, reasoning solution, code) data triples in this step. After conversion, they are used to train the pre-trained model called the code large model, with the following specific steps:

[0083] Feed the (question, inference solution) data pair into the pre-trained model ,Model According to the reasoning scheme, the corresponding code is generated. Then the (question, reasoning scheme, code) data triple is transformed into (question, reasoning scheme + code) data pair as training data. ,get .

[0084] Example 1

[0085] Experiments were conducted on the Human-Eval and MBPP datasets. To evaluate the effectiveness of this technology, we conducted experiments on the above datasets.

[0086] The experimental results are shown in the figure below:

[0087] Model Average Human-Eval MBPP Qwen2-7B-Instruct 65.52 77.44 53.60 Mistral-7B-V0.3-Instruct 39.7 37.80 41.60 Meta-Llama-3.1-8B-Instruct 62.02 64.63 59.40 Our Model <![CDATA[ 66.84 ]]> 79.27 ]]> ​ 54.40

[0088] The data in the table shows the pass@1 values ​​of different models on various datasets. Higher values ​​indicate better performance. The bolded data represent the best performing models. Underlined data indicate that our method outperforms traditional methods in this metric.

[0089] Experimental results show that our model outperforms traditional models in all tasks, fully demonstrating the effectiveness of the Monte Carlo tree search-based model self-evolution method. At the same time, our method has lower requirements on the quantity and quality of data.

[0090] Through our improved AutoMCTS method, this paper fully stimulates the reasoning ability of the large model, allowing the large model to improve autonomously, and ultimately obtains a large code model that can solve complex problems. It outperforms benchmark models such as Qwen2-7B-Instruct on multiple downstream tasks. It does not add additional parameters and does not require a large amount of high-quality fine-tuning data, avoiding the massive overhead of data acquisition. Compared with existing methods, it is lower in cost and better in effect, thus making a certain contribution to the development of large code models.

[0091] Functional features: The final product of training, the code model, can be used as a new large model in the code field. At the same time, the reasoning data and code data obtained from training can also provide support for other tasks.

[0092] Those skilled in the art can understand that the above embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope defined by the present application claims.

Claims

1. A code model self-evolution method based on Monte Carlo tree search, characterized in that: The method comprises: S1: Input the code generation task into the reasoning model in the form of a reasoning task, and generate reasoning steps according to the reasoning task; S2. Build an inference tree and obtain the optimal inference path and reward score based on the improved Monte Carlo tree search algorithm; S3. Self-evolution of the inference model based on the automatically generated inference and reward data according to the optimal inference path; S4. Based on the (problem, reasoning solution) data pairs generated by the improved Monte Carlo tree search algorithm, let the pre-trained model generate the code of the reasoning solution. The (problem, reasoning solution) data pairs are converted into (problem, reasoning solution + code) data pairs. The pre-trained model is trained using the (problem, reasoning solution + code) data pairs to obtain the code model. In step S1, the pre-trained model is used as the inference model to generate inference steps.

2. The code model self-evolution method according to claim 1, characterized in that: In step S2, the method for constructing the inference tree includes: The reasoning task is represented as a tree, in which each node represents a reasoning step, and each edge represents the reasoning process from the previous reasoning step to the next reasoning step. The root node is the description of the problem, and the leaf node is the final solution.

3. The code model self-evolution method according to claim 1, characterized in that: In step S2, an improved Monte Carlo tree search algorithm is used to search for the optimal path in the inference tree. The improvement is: For each expanded node, the reasoning model is used to generate reflections on the n+1th step for the problem-several step pairs. This is used to guide the specific generation of the n+1th step in the subsequent process, thereby alleviating the reasoning model's weakness in thinking about long contexts.

4. The code model self-evolution method according to claim 1, characterized in that: In step S2, an improved Monte Carlo tree search algorithm is used to search for the optimal path in the inference tree. The improvement is: Rewards and scoring are conducted for Step n. Scoring principles include: single-step correctness, consistency, and overall effectiveness of the solution.

5. The code model self-evolution method according to claim 4, characterized in that: In step S2, an improved Monte Carlo tree search algorithm is used to search for the optimal path in the inference tree. The improvement lies in: scoring the inference model according to the following principles: if there is an error in the inference step, score a is obtained; if there is no error in the inference step, but there is an error in the previous historical step, which may cause the subsequent step to go in the wrong direction, score b is obtained; if the inference step is correct and logically continuous with the previous step, and can solve the problem in the next one or more generations, score c is obtained; only when the current solution can completely solve the problem can score d be obtained, where a is the correct answer. <b<c<d。 6. The method for self-evolution of a large code model according to claim 4, characterized in that: In step S2, a quality score is assigned to each solution segment as follows: (1) Logic Verification: The model verifies the logical correctness of a single reasoning step and the continuity of the previous and subsequent steps. If the output does not meet expectations, the model records the low score of the path so that its priority can be reduced in subsequent MCTS iterations. (2) Performance evaluation: The performance of the solution is evaluated through time complexity analysis. If the generated solution is inefficient, the inference model will try to generate a more efficient solution through different inference paths. (3) Quality scoring: Based on the above evaluation, the inference model assigns a quality score to each solution to reflect the overall quality of the solution.

7. The code model self-evolution method according to claim 4, characterized in that: After an initial solution is generated and evaluated, the Monte Carlo Tree Search itself iterates through a feedback mechanism.

8. The code model self-evolution method according to claim 1, characterized in that: In step S3, the inference path generated by the Monte Carlo tree search algorithm is input into the inference model to enable it to self-evolve as follows: The reasoning path is used as input to train the reasoning model to learn to generate logically rigorous and clearly organized reasoning steps, so that this path is given priority in subsequent reasoning tasks.

9. The code model self-evolution method according to claim 1, characterized in that: Also includes: Before executing step S4, steps S2 and S3 are repeatedly executed to iterate and obtain the final inference model.

Images