Code generation model training method and device, electronic equipment and storage medium

Abstract

The invention discloses a code generation model training method and device, electronic equipment and a storage medium. A test case is obtained, and tree search is executed based on the test case; obtaining an evaluation value of each node in tree search, and performing backward transmission update according to the evaluation value of each node so as to update a state action value and an access count of each node; obtaining a correct search path in the tree search according to the state action value and the access count, and storing the correct search path in a model fine tuning database; and training a code generation model based on the model fine tuning database. Tree search is combined with a process reward model, pertinence and effectiveness of a search process are ensured, node evaluation and a backward transmission update mechanism enable the model to continuously optimize a state action value, and learning efficiency is enhanced. A code generation model is trained by a high-quality sample generated through reward guide tree search, and the capacity of the model for generating complex codes is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method and device for training a code generation model, an electronic device, and a storage medium. Background Art

[0002] In the self-training process of Large Language Model (LLM), the model generates content by processing massive amounts of data and optimizes its own performance in the process. This self-training method has the problem that the reasoning process is wrong but the result is correct, that is, the final answer generated may be correct, but some steps in the reasoning process are wrong. Due to the existence of these incorrect reasoning paths, the performance of LLM in complex tasks is difficult to be effectively improved. Summary of the invention

[0003] The present invention provides a method, device, electronic device and storage medium for training a code generation model.

[0004] According to a first aspect of the present disclosure, a method for training a code generation model is provided, comprising: Obtain a test case, and perform a tree search based on the test case; Obtaining the evaluation value of each node in the tree search, and performing backward transmission update according to the evaluation value of each node to update the state action value and access count of each node; Acquire a correct search path in the tree search according to the state action value and the access count, and store the correct search path in a model fine-tuning database; The code generation model is trained based on the model fine-tuning database.

[0005] Optionally, obtaining the evaluation value of each node in the tree search includes: Each node searched in the tree search is evaluated based on the process reward model to obtain an evaluation value of each node.

[0006] Optionally, the obtaining of the test case and performing a tree search based on the test case includes: Generate pseudo code based on the tree search thinking chain of the test case; Based on the pseudocode, an executable code of the test case is generated; wherein the pseudocode is used to control the granularity of the tree search behavior; In a case where the executable code satisfies the test case, determining that a result reward value of the executable code is not zero.

[0007] Optionally, the test case is generated by a result reward model; and the step of evaluating each node searched in the tree search based on the process reward model to obtain an evaluation value of each node further includes: Calculate a weighted sum according to the process reward value and the result reward value of each node to obtain the evaluation value of each node; Wherein, when the node is a leaf node, the weight coefficient of determining the result reward value is not zero.

[0008] Optionally, after the tree search is completed, performing a backward transmission update according to the evaluation value of each node to update the state action value and the access count of each node includes: A weighted average calculation is performed on at least one evaluation value of each of the nodes to obtain the state action value.

[0009] Optionally, after obtaining a correct search path in the tree search and storing the correct search path in a code generation model fine-tuning database as training data for the code generation model, the method further includes: All search paths are obtained in the tree search and stored in a process reward model training database in a preset format as training data for the process reward model; wherein the preset format is a test case, a search path, and an evaluation value of each node in the search path.

[0010] Optionally, before the code generation model performs a tree search based on the test case, the method further includes: The result reward model is subjected to preference training based on a preset preference data set; wherein the preset preference data set includes positive examples and negative examples; the positive examples include training test cases that meet the preferences of the preset preference data set; and the negative examples include training test cases that do not meet the preferences of the preset preference data set.

[0011] Optionally, before performing preference training on the result reward model based on a preset preference data set, the method further includes: Perform supervised fine-tuning training on the resulting reward model based on the training dataset.

[0012] Optionally, the training data in the training data set includes problems, solutions and test cases.

[0013] Optionally, the preset preference data set also includes instructions, questions, and codes.

[0014] Optionally, before obtaining all search paths in the tree search and storing them in a process reward model training database in a preset format as training data for the process reward model, the method further includes: The process reward model is pre-trained based on a pre-training data set; wherein the pre-training data set contains pre-training data of correct reasoning process and correct answer, incorrect reasoning process and correct answer, incorrect reasoning process and incorrect answer.

[0015] According to a second aspect of the present disclosure, a training device for a code generation model is provided, comprising: An execution unit, configured to obtain a test case and perform a tree search based on the test case; An evaluation unit, used to obtain an evaluation value of each node in the tree search, and to perform a backward transmission update according to the evaluation value of each node, so as to update the state action value and access count of each node; A first acquisition unit, configured to acquire a correct search path in the tree search according to the state action value and the access count, and store the correct search path in a model fine-tuning database; The first training unit is used to train the code generation model based on the model fine-tuning database.

[0016] Optionally, the execution unit is further used for: Each node searched in the tree search is evaluated based on the process reward model to obtain an evaluation value of each node.

[0017] Optionally, the execution unit includes: A first generation module, used for generating pseudo code based on the tree search thinking chain of the test case; A second generating module is used to generate executable code of the test case based on the pseudo code; wherein the pseudo code is used to control the granularity of the tree search behavior; The determination module is used to determine that the result reward value of the executable code is not zero when the executable code satisfies the test case.

[0018] Optionally, the test case is generated by a result reward model; and the evaluation unit is further used for: Calculate a weighted sum according to the process reward value and the result reward value of each node to obtain the evaluation value of each node; Wherein, when the node is a leaf node, the weight coefficient of determining the result reward value is not zero.

[0019] Optionally, the updating unit is further used for: A weighted average calculation is performed on at least one evaluation value of each of the nodes to obtain the state action value.

[0020] Optionally, the device further comprises: A second acquisition unit is used to obtain a correct search path in the tree search of the first acquisition unit, store the correct search path in the code generation model fine-tuning database as training data for the code generation model, obtain all search paths in the tree search, and store them in the process reward model training database in a preset format as training data for the process reward model; wherein the preset format is a test case, a search path, and an evaluation value of each node in the search path.

[0021] Optionally, the device further comprises: The second training unit is used to perform preference training on the result reward model based on a preset preference data set before the execution unit performs a tree search based on the test case in the code generation model; wherein the preset preference data set contains positive examples and negative examples; the positive examples contain training test cases that meet the preferences of the preset preference data set; and the negative examples contain training test cases that do not meet the preferences of the preset preference data set.

[0022] Optionally, the device further comprises: The training unit is further used to perform supervised fine-tuning training on the result reward model according to the training data set before performing preference training on the result reward model based on a preset preference data set.

[0023] Optionally, the training data in the training data set includes problems, solutions and test cases.

[0024] Optionally, the preset preference data set also includes instructions, questions and codes.

[0025] Optionally, the device further comprises: A third training unit is used to obtain all search paths in the tree search by the second acquisition unit, and store them in a process reward model training database in a preset format, so as to pre-train the process reward model based on a pre-training data set before using them as training data of the process reward model; wherein the pre-training data set contains pre-training data of correct reasoning process and correct answer, incorrect reasoning process and correct answer, and incorrect reasoning process and incorrect answer.

[0026] According to a third aspect of the present disclosure, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0027] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0028] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.

[0029] The training method, device, electronic device and storage medium of the code generation model provided by the present disclosure, the main technical scheme includes: obtaining test cases, and performing tree search based on the test cases; obtaining the evaluation value of each node in the tree search, and performing backward transmission update according to the evaluation value of each node to update the state action value and access count of each node; obtaining the correct search path in the tree search according to the state action value and the access count, and storing the correct search path in the model fine-tuning database; training the code generation model based on the model fine-tuning database. Compared with the related art, the embodiment of the present application combines the process reward model with the tree search to ensure the pertinence and effectiveness of the search process. The node evaluation and backward transmission update mechanism enable the model to continuously optimize the state action value and enhance the learning efficiency. The high-quality samples generated by the reward-guided tree search are used to self-train the code generation model, which improves the model's ability to generate complex code.

[0030] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Figure 1 A flowchart of a method for training a code generation model provided by an embodiment of the present disclosure; Figure 2 A schematic diagram of a Monte Carlo tree search process provided by an embodiment of the present disclosure; Figure 3 A schematic diagram of the structure of a training device for a code generation model provided in an embodiment of the present disclosure; Figure 4 A schematic diagram of the structure of a training device for a code generation model provided in an embodiment of the present disclosure; Figure 5 A schematic block diagram of an exemplary electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] The following describes the code generation model training method, device, electronic device and storage medium of the embodiments of the present disclosure with reference to the accompanying drawings.

[0034] Figure 1 A flowchart of a method for training a code generation model provided in an embodiment of the present disclosure.

[0035] like Figure 1 As shown, the method comprises the following steps: Step 101: obtain a test case, and perform a tree search based on the test case.

[0036] Build a test case execution tree where each node represents the state of a test case and the branches of the tree search represent different test paths or decision points.

[0037] In some embodiments, during the tree search process, during the i-th tree search, the process starts from Start, representing the initial state containing the test cases. The algorithm then explores the tree by moving to nodes selected by the algorithm. , the algorithm is:

[0038] in, is the value corresponding to the state-action, is the number of times the child node is visited (which will be updated during the search process), is the number of times a's parent node is visited. This selection process continues until the leaf node. It is a weight factor used to balance exploration and exploitation.

[0039] Step 102, obtaining the evaluation value of each node in the tree search, and performing a backward transmission update according to the evaluation value of each node to update the state action value and the access count of each node.

[0040] After the tree search is completed, the evaluation value of each node is used to pass the update backward to optimize the performance of the search algorithm and ensure that the algorithm can learn and improve from past search experience.

[0041] First, we need to determine the path from the leaf node where the tree search ends to the root node. Each node in the path needs to be updated. For each node on the path, we increase its visit count, which is an indicator that records the number of times a node has been visited.

[0042] Starting from a leaf node, accumulate the total reward from that node to the final state. This total reward will be used to update the state-action value of each node on the path.

[0043] The state-action value of each node is calculated by taking a weighted average of the results of multiple simulations of different nodes on each path, where the weight is determined by the ratio of the simulation result to the total number of simulations each time, so that more experience can be accumulated in the strategy tree, thereby more accurately estimating the potential value of different state-action pairs.

[0044] Step 103: Obtain a correct search path in the tree search according to the state action value and the access count, and store the correct search path in a model fine-tuning database.

[0045] The state-action value reflects the expected benefit of taking an action in a specific state, and the access count records the frequency of exploration of the state-action combination. Specifically, through a careful evaluation of the state-action value, combined with the exploration depth reflected by the access count, the search algorithm gradually traverses the tree structure, starting from the root node, and deeply explores along the branch with the most potential (i.e., the best state-action value) until the correct search path that meets specific conditions is found. This path represents the optimal or better action sequence from the starting state to the target state under the current information and rules.

[0046] The correct path and the corresponding test case are put into the fine-tuning database as a new data set and added to the fine-tuning process of the code generation model.

[0047] Step 104: training the code generation model based on the model fine-tuning database.

[0048] In some embodiments, the code generation model is a large language model.

[0049] By fine-tuning the correct tree search path in the model database, the model can continuously correct itself based on the reward value feedback of the nodes in the search path, thereby performing self-optimization.

[0050] The training method of the code generation model provided by the present disclosure includes the following main technical solutions: obtaining test cases, performing tree search based on the test cases; obtaining the evaluation value of each node in the tree search, and performing backward transmission update according to the evaluation value of each node to update the state action value and access count of each node; obtaining the correct search path in the tree search according to the state action value and the access count, and storing the correct search path in the model fine-tuning database; training the code generation model based on the model fine-tuning database. Compared with the related art, the embodiment of the present application combines the process reward model with the tree search to ensure the pertinence and effectiveness of the search process. The node evaluation and backward transmission update mechanism enable the model to continuously optimize the state action value and enhance the learning efficiency. The high-quality samples generated by the reward-guided tree search are used to self-train the code generation model, which improves the model's ability to generate complex code.

[0051] In some embodiments, during the search process, when a search path is explored from the root node to the leaf node to form a complete tree search process, that is, a partial reasoning process in the tree search, the partial reasoning process can be determined as a prompt for further node expansion. In an embodiment of the present application, the code generation model is a large language model, which is generated by sampling at a higher temperature to ensure the diversity of the explored nodes.

[0052] In some embodiments, the trained code generation model can be used to assist in code development, for example: During the code development phase, it provides code completion functions, such as suggesting variable names, functions, syntax structures, etc. It automatically detects potential errors in the code, such as suggesting or directly fixing common logic problems, syntax errors, or performance bottlenecks.

[0053] At the code testing node, it provides code optimization suggestions, such as improving algorithm efficiency, reducing redundant code, etc.; it generates unit test code to assist developers in verifying program behavior under different input conditions.

[0054] In some embodiments, when determining the evaluation values ​​of different nodes in step 102, the following steps may also be performed: Each node searched in the tree search is evaluated based on the process reward model to obtain an evaluation value of each node.

[0055] Process Reward Models (PRM) is an evaluation mechanism that assigns an evaluation value to each node in the tree search process, and allocates it based on comprehensive considerations such as goal achievement, resource consumption, and uncertainty reduction. In some embodiments, the reward model can be a simple linear function or a complex nonlinear function. Specifically, it is set according to actual needs, and the embodiments of the present application are not limited to this.

[0056] In some embodiments, the next node to be executed is selected based on the evaluation values ​​of different nodes. A greedy strategy can be used to select the node with the highest evaluation value, or an exploration-utilization strategy (such as the UCB1 algorithm) can be used to balance the exploration of unknown nodes and the use of known information. It should be noted that this narrative method is only an exemplary description and is not a specific limitation on the specific selection strategy. The embodiments of the present application do not limit the algorithm for selecting the next node.

[0057] In some embodiments, step 101 further includes: Generate pseudo code based on the tree search thinking chain of the test case.

[0058] When generating pseudocode, first, clarify the basic process of tree search, including selecting nodes, expanding nodes, simulating and returning evaluation values. Secondly, describe how to select the next node to be processed from the current node. For example, compare the evaluation values ​​or visit times of different nodes. Again, explain how to generate the child nodes of a node after selecting a node, that is, explore all possible actions of the node. Finally, explain how to simulate the results of the problem from the selected node until a terminal state is reached. It should be noted that this narrative method is only an exemplary description, and is not a specific limitation on the specific pseudocode content, and the embodiments of the present application do not limit this.

[0059] Based on the pseudocode, an executable code of the test case is generated; wherein the pseudocode is used to control the granularity of the tree search behavior.

[0060] According to the granularity of the reasoning steps controlled by the pseudocode, each step in the pseudocode is converted into a specific programming language instruction to define the data structure.

[0061] In a case where the executable code satisfies the test case, determining that a result reward value of the executable code is not zero.

[0062] In the process of tree search, when a leaf node is found, it means that an executable code is generated. The search path from the root node to the leaf node is the generation path of the executable code. However, the executable code may not meet the test case, so it is marked with the result reward value. In actual application, the specific value of the reward result value is determined according to the numerical range of the reward value. For example, if the interval of the reward result value is 1 and 0, that is, if the executable code meets the test case, the reward result value is 1, and if it does not meet the test case, the reward result value is 0.

[0063] When the generated executable code cannot meet the requirements of the test case, it means that the generation path of the executable code is not the correct path, and the result reward value is set to 0; when the generated executable code can meet the requirements of the test case, it means that the generation path of the executable code is the correct path, and the result reward value is set to 1.

[0064] In some embodiments, the code generation model executes the search tree according to the test case, that is, the process of generating executable code. In order to better control the granularity of the reasoning steps, the code generation model first generates a complete chain of thoughts, and then generates executable code step by step according to the chain of thoughts. First, a detailed pseudocode is generated, and then the pseudocode is used for guidance to generate the final executable code. In addition to adjusting the level of detail of the pseudocode to control the granularity of the reasoning behavior, this method can also realize the generation of different specific codes under the same pseudocode.

[0065] Pseudocode can better adjust and optimize the granularity of reasoning steps in the search process, ensuring the accuracy and efficiency of the search. When the executable code meets the test case, the reward value is not zero, which helps to strengthen the positive feedback in the learning process, thereby improving the learning effect and decision quality of the code generation model.

[0066] In some embodiments, the test case is generated by a result reward model; and the step of evaluating each node searched in the tree search based on the process reward model to obtain an evaluation value of each node further includes: A weighted sum is calculated according to the process reward value of each node and the result reward value generated by the Outcome Reward Models (ORM), so as to obtain the evaluation value of each node.

[0067] Wherein, when the node is a leaf node, the weight coefficient of determining the result reward value is not zero.

[0068] In some embodiments, the result reward value of the executable code is assigned to the corresponding leaf node as the evaluation value of the leaf node, and the evaluation value of the leaf node can be used to determine whether the search path from the leaf node to the root node is the correct path. Specifically, the relative importance of the process reward value and the result reward value in the total evaluation value is determined by setting a weight coefficient. For each node, the process reward value and the result reward value are weighted and summed according to their respective weights, and the calculation formula is:

[0069] in, is the process reward value, is the result reward value, is the weight factor, when the node is a leaf node ,otherwise .

[0070] That is, the evaluation value of the leaf node is the result reward value, and the evaluation value of other nodes is the process reward value. The calculated weighted sum is used as the evaluation value of the node, and the information of the corresponding node in the tree is updated.

[0071] In some embodiments, when calculating the evaluation value, a time discount factor may also be considered to ensure the long-term consistency of the evaluation value.

[0072] By assigning non-zero weights to the leaf node result reward values, the search algorithm is guided to prefer paths that lead to good final results, enabling the algorithm to make more reasonable and effective decisions in a dynamic environment, thereby improving the efficiency and success rate of the overall decision-making system.

[0073] Optionally, after the tree search is completed, performing a backward transmission update according to the evaluation value of each node to update the state action value and the access count of each node includes: A weighted average calculation is performed on at least one evaluation value of each of the nodes to obtain the state action value.

[0074] At the end of the i-th simulation, along the leaf node Each edge (s, a) on the path to the root undergoes one backward pass of update. Updates to its state-action value and visit count are performed according to the following rules: ,

[0075] in, is the node access count, is the state action value.

[0076] The denominator of the formula is The numerator is the sum of the evaluation values ​​of all simulation results. The state-action value is calculated by taking a weighted average of the results of multiple simulations, where the weight is determined according to the ratio of the result of each simulation to the total number of simulations.

[0077] In some embodiments, after obtaining a correct search path in the tree search and storing the correct search path in a code generation model fine-tuning database as training data for the code generation model, the method further includes: All search paths are obtained in the tree search and stored in a process reward model training database in a preset format as training data for the process reward model; wherein the preset format is a test case, a search path, and an evaluation value of each node in the search path.

[0078] In some embodiments, the process reward model is pre-trained based on a pre-training data set; wherein the pre-training data set includes pre-training data of correct reasoning process and correct answer, incorrect reasoning process and correct answer, and incorrect reasoning process and incorrect answer.

[0079] All problem-solving processes should be (Q is the test case, s is the exploration path, and v is the evaluation value corresponding to this part of the reasoning process) format is processed to serve as the training data of the process reward model for further training of the process reward model; the initial value model is obtained by fine-tuning (SFT) on the basic model through the training database to guide the LLM tree to search for higher quality solutions. The data set includes multiple data such as correct reasoning process-correct answer, incorrect reasoning process-correct answer, and incorrect reasoning process-incorrect answer.

[0080] The process reward model can determine the reward value based on the specific steps and intermediate results of the task, which makes the decision-making process of the model more explainable. In the code generation task, the reward value can be used to determine why a certain path received a higher reward at a certain step, thereby improving the transparency and traceability of the model-generated code.

[0081] By obtaining all search paths of the tree search and storing them in a preset format, the training data of the process reward model is effectively accumulated, ensuring that the model can learn the evaluation values ​​of different search paths, thereby improving the prediction accuracy and generalization ability of the model in complex decision-making scenarios, and providing data support for optimizing the search algorithm and improving decision quality.

[0082] In some embodiments, the code generation model is self-trained through tree search guidance, so that the code generation model can continuously self-correct according to the reward feedback in the tree search. When a potential valid code fragment is searched, the reward value guidance of the process reward model can prompt the search to go deeper. When the search fails or invalid code is generated, the code generation model will adjust the search strategy and generation method according to the feedback of the process reward, so as to self-optimize.

[0083] In some embodiments, before the code generation model performs a tree search based on the test case, the method further includes: The result reward model is subjected to preference training based on a preset preference data set; wherein the preset preference data set includes positive examples and negative examples; the positive examples include training test cases that meet the preferences of the preset preference data set; and the negative examples include training test cases that do not meet the preferences of the preset preference data set.

[0084] In some embodiments, the preset preference data set also includes instructions, questions and codes, as well as positive examples (test cases that meet the preferences) and negative examples (test cases that are inconsistent with the preferences). The construction rules of the preference data set are as follows: for positive examples, three completely matching test cases are selected; for negative examples, the outputs of the three sampled test cases are shuffled, and the original inputs are connected to generate incompletely matching input-output pairs, thereby constructing test cases that do not meet the preferences.

[0085] By training the result reward model based on a preset preference data set, the model can learn to distinguish between examples that meet and do not meet the preferences, improve the accuracy of its preference decisions in practical applications, and ensure that the output results are closer to user expectations.

[0086] In some embodiments, before performing preference training on the result reward model based on a preset preference data set, the method further includes: Perform supervised fine-tuning training on the resulting reward model based on the training dataset.

[0087] The core goal of supervised fine-tuning training is to ensure that the output generated by the result reward model follows a predetermined format so that the generated test cases can be accurately parsed and extracted. The training data mainly comes from the TACO dataset. In some embodiments, the training dataset includes: problems, solutions, and test cases.

[0088] In some embodiments, the tree search algorithm is a Monte Carlo Tree Search (MCTS), see Figure 2 , Figure 2 A schematic diagram of a Monte Carlo tree search process provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the cylinder above the figure represents a data set, which contains a series of Yes, among them For the test case problem, The following is based on The decision sequence of arrive According to the strategy The decisions generated under different conditions eventually form a , , , …, ) dataset.

[0089] Please continue reading Figure 2 , Figure 2 In ORM, the above dataset The information in the left column is used as input. The PRM and Policy Model receive the data set obtained by the MCTS model on the left column as input. The output of these modules is the MCTS tree structure on the right column. The structure of the MCTS tree contains multiple nodes. White nodes represent correct nodes, and black nodes represent incorrect nodes. MCTS searches for the optimal decision path by continuously expanding and evaluating this tree structure.

[0090] The MCTS-based search framework has strong scalability in large-scale problems; as the task scale increases, tree search can gradually expand the search space, and the process reward model provides guidance during the search process to ensure that the search focuses on the path most likely to succeed, thereby avoiding inefficiency caused by an overly large search space.

[0091] Corresponding to the above-mentioned training method of the code generation model, the present invention also proposes a training device for the code generation model. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, the details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.

[0092] Figure 3 A schematic diagram of a code generation model training device provided by an embodiment of the present disclosure, such as Figure 3 As shown, it includes: According to the second aspect of the present disclosure, a training device for a code generation model is provided, including: An execution unit 21, configured to obtain a test case and perform a tree search based on the test case; An evaluation unit 22, used to obtain an evaluation value of each node in the tree search, and to perform a backward transmission update according to the evaluation value of each node, so as to update the state action value and access count of each node; A first acquisition unit 23, configured to acquire a correct search path in the tree search according to the state action value and the access count, and store the correct search path in a model fine-tuning database; The first training unit 24 is used to train the code generation model based on the model fine-tuning database.

[0093] The training device of the code generation model provided by the present disclosure has the following main technical solutions: obtaining test cases and performing tree search based on the test cases; obtaining the evaluation value of each node in the tree search, and performing backward transmission update according to the evaluation value of each node to update the state action value and access count of each node; obtaining the correct search path in the tree search according to the state action value and the access count, and storing the correct search path in the model fine-tuning database; training the code generation model based on the model fine-tuning database. Compared with the related art, the embodiment of the present application combines the process reward model with the tree search to ensure the pertinence and effectiveness of the search process. The node evaluation and backward transmission update mechanism enable the model to continuously optimize the state action value and enhance the learning efficiency. The high-quality samples generated by the reward-guided tree search are used to self-train the code generation model, which improves the model's ability to generate complex code.

[0094] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the execution unit 21 is also used for: Each node searched in the tree search is evaluated based on the process reward model to obtain an evaluation value of each node.

[0095] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the execution unit 21 includes: A first generating module 211 is used to generate pseudo code based on the tree search thinking chain of the test case; A second generating module 212 is used to generate executable code of the test case based on the pseudo code; wherein the pseudo code is used to control the granularity of the tree search behavior; The determination module 213 is configured to determine that the result reward value of the executable code is not zero when the executable code satisfies the test case.

[0096] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the test case is generated by the result reward model; the evaluation unit 22 is also used for: Calculate a weighted sum according to the process reward value and the result reward value of each node to obtain the evaluation value of each node; Wherein, when the node is a leaf node, the weight coefficient of determining the result reward value is not zero.

[0097] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the first acquisition unit 23 is also used for: A weighted average calculation is performed on at least one evaluation value of each of the nodes to obtain the state action value.

[0098] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the device also includes: The second acquisition unit 25 is used to obtain the correct search path in the tree search of the first training unit 24, store the correct search path in the code generation model fine-tuning database as training data for the code generation model, obtain all search paths in the tree search, and store them in the process reward model training database in a preset format as training data for the process reward model; wherein the preset format is test cases, search paths, and evaluation values ​​of each node in the search path.

[0099] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the device also includes: The second training unit 26 is used to perform preference training on the result reward model based on a preset preference data set before the execution unit 21 performs a tree search based on the test case in the code generation model; wherein the preset preference data set includes positive examples and negative examples; the positive examples include training test cases that meet the preferences of the preset preference data set; and the negative examples include training test cases that do not meet the preferences of the preset preference data set.

[0100] Furthermore, in a possible implementation of the embodiment of the present disclosure, the device further includes: The training unit is further used to perform supervised fine-tuning training on the result reward model according to the training data set before performing preference training on the result reward model based on a preset preference data set.

[0101] Furthermore, in a possible implementation of the embodiment of the present disclosure, the training data in the training data set includes problems, solutions and test cases.

[0102] Furthermore, in a possible implementation of the embodiment of the present disclosure, the preset preference data set also includes instructions, questions and codes.

[0103] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the device also includes: The third training unit 27 is used to obtain all search paths in the tree search in the second acquisition unit 25, and store them in the process reward model training database in a preset format, so as to pre-train the process reward model based on a pre-training data set before using them as training data of the process reward model; wherein the pre-training data set contains pre-training data of correct reasoning process and correct answer, incorrect reasoning process and correct answer, and incorrect reasoning process and incorrect answer.

[0104] It should be noted that the above explanation of the method embodiment is also applicable to the device of the embodiment of the present disclosure, and the principle is the same, which is no longer limited in the embodiment of the present disclosure.

[0105] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0106] Figure 5 A schematic block diagram of an example electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0107] like Figure 5 As shown, the device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 302 or a computer program loaded from a storage unit 308 to a RAM (Random Access Memory) 303. In the RAM 303, various programs and data required for the operation of the device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O (Input / Output) interface 305 is also connected to the bus 304.

[0108] A number of components in the device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, CPU (Central Processing Unit), GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as the training method of the code generation model. For example, in some embodiments, the training method of the code generation model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to execute the aforementioned training method of the code generation model in any other appropriate manner (for example, by means of firmware).

[0110] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0112] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0114] The systems and techniques described herein may be implemented in a computing system that includes a backend component (e.g., as a data server), or a computing system that includes a middleware component (e.g., an application server), or a computing system that includes a frontend component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0115] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0116] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a code generation model, characterized in that: include: Obtain a test case, and perform a tree search based on the test case; Obtaining the evaluation value of each node in the tree search, and performing backward transmission update according to the evaluation value of each node to update the state action value and access count of each node; Acquire a correct search path in the tree search according to the state action value and the access count, and store the correct search path in a model fine-tuning database; The code generation model is trained based on the model fine-tuning database.

2. The method according to claim 1, characterized in that: The evaluation value of each node in the tree search is obtained by: Each node searched in the tree search is evaluated based on the process reward model to obtain an evaluation value of each node.

3. The method according to claim 1, characterized in that The obtaining of the test case and performing a tree search based on the test case comprises: Generate pseudo code based on the tree search thinking chain of the test case; Based on the pseudocode, an executable code of the test case is generated; wherein the pseudocode is used to control the granularity of the tree search behavior; In a case where the executable code satisfies the test case, determining that a result reward value of the executable code is not zero.

4. The method according to claim 2, characterized in that: The test case is generated by the result reward model; The process reward model is used to evaluate each node found in the tree search to obtain an evaluation value of each node, and the evaluation value of each node also includes: Calculate a weighted sum according to the process reward value and the result reward value of each node to obtain the evaluation value of each node; Wherein, when the node is a leaf node, the weight coefficient of determining the result reward value is not zero.

5. The method according to claim 4, characterized in that After the tree search is completed, the update is transmitted backward according to the evaluation value of each node to update the state action value and the access count of each node, including: A weighted average calculation is performed on at least one evaluation value of each of the nodes to obtain the state action value.

6. The method according to claim 1, characterized in that After obtaining a correct search path in the tree search and storing the correct search path in a code generation model fine-tuning database as training data for the code generation model, the method further includes: All search paths are obtained in the tree search and stored in a process reward model training database in a preset format as training data for the process reward model; wherein the preset format is a test case, a search path, and an evaluation value of each node in the search path.

7. The method according to claim 4, characterized in that The code generation model further includes: performing a tree search based on a test case: The result reward model is subjected to preference training based on a preset preference data set; wherein the preset preference data set includes positive examples and negative examples; the positive examples include training test cases that meet the preferences of the preset preference data set; and the negative examples include training test cases that do not meet the preferences of the preset preference data set.

8. The method according to claim 7, characterized in that Before performing preference training on the result reward model based on the preset preference data set, the method further includes: Perform supervised fine-tuning training on the resulting reward model based on the training dataset.

9. The method according to claim 8, characterized in that The training data in the training data set includes problems, solutions and test cases.

10. The method according to claim 7, characterized in that The preset preference data set also includes instructions, questions and codes.

11. The method according to claim 6, characterized in that Before obtaining all search paths in the tree search and storing them in a process reward model training database in a preset format as training data for the process reward model, the method further includes: The process reward model is pre-trained based on a pre-training data set; wherein the pre-training data set contains pre-training data of correct reasoning process and correct answer, incorrect reasoning process and correct answer, incorrect reasoning process and incorrect answer.

12. A training device for a code generation model, characterized in that: include: An execution unit, configured to obtain a test case and perform a tree search based on the test case; An evaluation unit, used to obtain an evaluation value of each node in the tree search, and to perform a backward transmission update according to the evaluation value of each node, so as to update the state action value and access count of each node; A first acquisition unit, configured to acquire a correct search path in the tree search according to the state action value and the access count, and store the correct search path in a model fine-tuning database; A training unit is used to train the code generation model based on the model fine-tuning database.

13. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

15. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images