Training Method and Device for Code Generation Model, Electronic Device, and Storage Medium

Abstract

The present disclosure discloses a training method and apparatus for a code generation model, an electronic device, and a storage medium. Test cases are obtained, and tree search is performed based on the test cases; evaluation values of each node in the tree search are obtained, and backward propagation updates are performed according to the evaluation values of each node to update the state-action values and access counts of each node; the correct search path is obtained in the tree search according to the state-action values and the access counts, and the correct search path is stored in the model fine-tuning database; the code generation model is trained based on the model fine-tuning database. By combining tree search with a process reward model, the pertinence and effectiveness of the search process are ensured. The node evaluation and backward propagation update mechanism enable the model to continuously optimize the state-action values and enhance the learning efficiency. By using high-quality samples generated by reward-guided tree search to self-train the code generation model, the ability of the model to generate complex code is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a training method and apparatus for a code generation model, an electronic device, and a storage medium. Background Art

[0002] In the self-training process of a large language model (LLM), the model processes a large amount of data to generate content and optimizes its own performance in this process. There is a situation where the reasoning process is incorrect but the result is correct in this self-training method, that is, the finally generated answer may be correct, but some steps in the reasoning process are incorrect. Due to the existence of these incorrect reasoning paths, it is difficult to effectively improve the performance of the LLM in complex tasks. Summary of the Invention

[0003] The present disclosure provides a training method, apparatus, electronic device, and storage medium for a code generation model.

[0004] According to a first aspect of the present disclosure, there is provided a training method for a code generation model, including:

[0005] Obtain test cases and perform tree search based on the test cases;

[0006] Obtain evaluation values of each node in the tree search, and perform backward propagation update according to the evaluation values of each node to update the state-action values and access counts of each node;

[0007] Obtain a correct search path in the tree search according to the state-action values and the access counts, and store the correct search path in a model fine-tuning database;

[0008] Train a code generation model based on the model fine-tuning database.

[0009] Optionally, the obtaining of the evaluation values of each node in the tree search includes:

[0010] Evaluate each node searched in the tree search based on a process reward model to obtain the evaluation values of each node.

[0011] Optionally, the obtaining of the test cases and performing tree search based on the test cases includes:

[0012] Generate pseudo-code based on the tree search thought chain of the test cases;

[0013] Generate executable code for the test cases based on the pseudo-code; wherein the pseudo-code is used to control the behavior granularity of the tree search;

[0014] When the executable code meets the test case, determine that the result reward value of the executable code is not zero.

[0015] Optionally, the test case is generated by a result reward model; the evaluating each node searched in the tree search based on a process reward model to obtain the evaluation value of each node further includes:

[0016] Calculate the weighted sum according to the process reward value and the result reward value of each node, and calculate the evaluation value of each node;

[0017] Wherein, when the node is a leaf node, determine that the weight coefficient of the result reward value is not zero.

[0018] Optionally, after the tree search ends, performing backward propagation update according to the evaluation value of each node to update the state-action value and the visit count of each node includes:

[0019] Perform weighted average calculation on at least one evaluation value of each node respectively to obtain the state-action value.

[0020] Optionally, after obtaining the correct search path in the tree search and storing the correct search path in the code generation model fine-tuning database as the training data of the code generation model, the method further includes:

[0021] Obtain all search paths in the tree search and store them in the process reward model training database in a preset format as the training data of the process reward model; wherein, the preset format is test case, search path, and evaluation value of each node in the search path.

[0022] Optionally, before the code generation model performs tree search based on the test case, the method further includes:

[0023] Perform preference training on the result reward model based on a preset preference dataset; wherein, the preset preference dataset includes positive examples and negative examples; the positive examples include training test cases that conform to the preferences of the preset preference dataset; the negative examples include training test cases that do not conform to the preferences of the preset preference dataset.

[0024] Optionally, before performing preference training on the result reward model based on a preset preference dataset, the method further includes:

[0025] Perform supervised fine-tuning training on the result reward model according to the training dataset.

[0026] Optionally, the training data in the training dataset includes questions, solutions, and test cases.

[0027] Optionally, the preset preference data set further includes instructions, questions, and codes.

[0028] Optionally, before obtaining all search paths in the tree search and storing them in the process reward model training database in a preset format as the training data of the process reward model, the method further includes:

[0029] Pre-training the process reward model based on a pre-training data set; wherein the pre-training data set contains pre-training data of correct reasoning processes and correct answers, incorrect reasoning processes and correct answers, and incorrect reasoning processes and incorrect answers.

[0030] According to a second aspect of the present disclosure, there is provided a training device for a code generation model, including:

[0031] An execution unit, configured to obtain a test case and perform a tree search based on the test case;

[0032] An evaluation unit, configured to obtain evaluation values of each node in the tree search, and perform backward propagation update according to the evaluation values of each node to update the state-action values and access counts of each node;

[0033] A first acquisition unit, configured to obtain a correct search path in the tree search according to the state-action value and the access count, and store the correct search path in a model fine-tuning database;

[0034] A first training unit, configured to train a code generation model based on the model fine-tuning database.

[0035] Optionally, the execution unit is further configured to:

[0036] Evaluate each node searched in the tree search based on a process reward model to obtain evaluation values of each node.

[0037] Optionally, the execution unit includes:

[0038] A first generation module, configured to generate pseudo-code based on the tree search thought chain of the test case;

[0039] A second generation module, configured to generate executable code of the test case based on the pseudo-code; wherein the pseudo-code is used to control the behavior granularity of the tree search;

[0040] A determination module, configured to determine that the result reward value of the executable code is not zero when the executable code meets the test case.

[0041] Optionally, the test case is generated by the result reward model; the evaluation unit is further configured to:

[0042] Calculate the weighted sum based on the process reward values and the result reward value of each of the nodes, and calculate the evaluation value of each of the nodes;

[0043] Wherein, when the node is a leaf node, it is determined that the weight coefficient of the result reward value is not zero.

[0044] Optionally, the update unit is further configured to:

[0045] Perform weighted average calculation on at least one evaluation value of each of the nodes respectively to obtain the state-action value.

[0046] Optionally, the apparatus further includes:

[0047] A second acquisition unit, configured to, after obtaining the correct search path in the tree search by the first acquisition unit, store the correct search path into the code generation model fine-tuning database as the training data of the code generation model, and then obtain all search paths in the tree search and store them into the process reward model training database in a preset format as the training data of the process reward model; wherein, the preset format is the test case, the search path, and the evaluation value of each node in the search path.

[0048] Optionally, the apparatus further includes:

[0049] A second training unit, configured to perform preference training on the result reward model based on a preset preference dataset before the execution unit performs tree search based on the test case in the code generation model; wherein, the preset preference dataset includes positive examples and negative examples; the positive examples include training test cases that conform to the preferences of the preset preference dataset; the negative examples include training test cases that do not conform to the preferences of the preset preference dataset.

[0050] Optionally, the apparatus further includes:

[0051] The training unit is further configured to perform supervised fine-tuning training on the result reward model according to the training dataset before performing preference training on the result reward model based on the preset preference dataset.

[0052] Optionally, the training data in the training dataset includes questions, solutions, and test cases.

[0053] Optionally, the preset preference dataset further includes instructions, questions, and codes.

[0054] Optionally, the apparatus further includes:

[0055] A third training unit, configured to pre-train the process reward model based on a pre-training dataset before the second obtaining unit obtains all search paths in the tree search and stores them in the process reward model training database in a preset format as training data for the process reward model; wherein, the pre-training dataset includes pre-training data of correct inference processes and correct answers, incorrect inference processes and correct answers, and incorrect inference processes and incorrect answers.

[0056] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0057] At least one processor; and

[0058] A memory communicatively connected to the at least one processor; wherein,

[0059] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the foregoing first aspect.

[0060] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in the foregoing first aspect.

[0061] According to a fifth aspect of the present disclosure, there is provided a computer program product, including a computer program, where the computer program implements the method described in the foregoing first aspect when executed by a processor.

[0062] The training method, device, electronic device, and storage medium for the code generation model provided by the present disclosure mainly include the following technical solutions: obtaining a test case and performing a tree search based on the test case; obtaining evaluation values of each node in the tree search, and performing backward propagation update according to the evaluation values of each node to update the state-action values and access counts of each node; obtaining a correct search path in the tree search according to the state-action values and the access counts, and storing the correct search path in the model fine-tuning database; training the code generation model based on the model fine-tuning database. Compared with the related art, the embodiments of the present application ensure the pertinence and effectiveness of the search process through the combination of tree search and process reward model, and the node evaluation and backward propagation update mechanism enable the model to continuously optimize the state-action values and enhance the learning efficiency. The code generation model is self-trained with high-quality samples generated by reward-guided tree search, improving the model's ability to generate complex code.

[0063] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become readily understood through the following description. Description of the Drawings

[0064] The drawings are used to better understand the present solution and do not constitute a limitation to the present disclosure. Among them:

[0065] Figure 1 is a schematic flowchart of a method for training a code generation model provided by an embodiment of the present disclosure;

[0066] Figure 2 is a schematic flowchart of a Monte Carlo tree search provided by an embodiment of the present disclosure;

[0067] Figure 3 is a schematic structural diagram of a device for training a code generation model provided by an embodiment of the present disclosure;

[0068] Figure 4 is a schematic structural diagram of a device for training a code generation model provided by an embodiment of the present disclosure;

[0069] Figure 5 is a schematic block diagram of an exemplary electronic device provided by an embodiment of the present disclosure. Detailed Embodiments

[0070] The following describes exemplary embodiments of the present disclosure with reference to the drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted below.

[0071] The following describes a method, device, electronic device, and storage medium for training a code generation model according to embodiments of the present disclosure with reference to the drawings.

[0072] Figure 1 is a schematic flowchart of a method for training a code generation model provided by an embodiment of the present disclosure.

[0073] As Figure 1 shown, the method includes the following steps:

[0074] Step 101, obtain a test case and perform tree search based on the test case.

[0075] Construct a test case execution tree, where each node represents the state of a test case, and the branches of the tree search represent different test paths or decision points.

[0076] In some embodiments, during the tree search process, in the i-th tree search, this process starts from which represents the initial state containing the test case. Then, the algorithm explores the tree by selecting nodes according to the algorithm and the algorithm is as follows:

[0077]

[0078] where is the value of the state-action correspondence, is the number of times the child node is visited (which will be updated during the search process), is the number of times the parent node of a is visited. This selection process loops until the leaf node, is the weight factor used to balance the weights between exploration and exploitation.

[0079] Step 102: Obtain the evaluation values of each node in the tree search, and perform backward propagation update according to the evaluation values of each node to update the state-action values and access counts of each node. [[ID=th=23]]

[0080] After the tree search ends, perform backward propagation update according to the evaluation values of each node to optimize the performance of the search algorithm and ensure that the algorithm can learn and improve from past search experiences.

[0081] First, it is necessary to determine the path from the leaf node where the tree search ends to the root node, and each node in the path needs to be updated. For each node on the path, increase its access count, and the access count is an indicator that records the number of times the node is visited.

[0082] Starting from the leaf node, accumulate the total reward from this node to the final state. This total reward will be used to update the state-action values of each node on the path.

[0083] Calculate the state-action value of each node by weighted average calculation of the results of multiple simulations of different nodes on each path, where the weights are determined according to the ratio of the results of each simulation to the total number of simulations, so as to accumulate more experience in the policy tree and more accurately estimate the potential values of different state-action pairs.

[0084] Step 103: Obtain the correct search path in the tree search according to the state-action value and the access count, and store the correct search path in the model fine-tuning database.

[0085] The state-action value reflects the expected reward that can be obtained by taking a certain action in a specific state, and the visit count records the frequency of exploration of the state-action combination. Specifically, through a detailed evaluation of the state-action value, combined with the exploration depth reflected by the visit count, based on the search algorithm, gradually traverse in the tree structure, starting from the root node, and deeply explore along the most promising (i.e., the optimal state-action value) branch until the correct search path that meets specific conditions is found. This path represents the optimal or sub-optimal action sequence from the starting state to the target state under the current information and rules.

[0086] Put the correct path and the corresponding test cases into the fine-tuning database as a new data set and add them to the fine-tuning process of the code generation model.

[0087] Step 104, train the code generation model based on the model fine-tuning database.

[0088] In some embodiments, the code generation model is a large language model.

[0089] Through the correct tree search path in the model fine-tuning database, the model can continuously self-correct according to the reward value feedback of the nodes in the search path, so as to perform self-optimization.

[0090] The training method of the code generation model provided by the present disclosure mainly includes the following technical solutions: obtaining test cases, performing tree search based on the test cases; obtaining the evaluation values of each node in the tree search, and performing backward propagation update according to the evaluation values of each node to update the state-action value and visit count of each node; obtaining the correct search path in the tree search according to the state-action value and the visit count, and storing the correct search path in the model fine-tuning database; training the code generation model based on the model fine-tuning database. Compared with the related art, the embodiment of the present application combines tree search with a process reward model to ensure the pertinence and effectiveness of the search process. The node evaluation and backward propagation update mechanism enables the model to continuously optimize the state-action value and enhance the learning efficiency. Self-training the code generation model with high-quality samples generated by reward-guided tree search improves the ability of the model to generate complex code.

[0091] In some embodiments, during the search process, when a search path explores from the root node to the leaf node to form a complete tree search process, that is, after a partial inference process in the tree search, the partial inference process can be determined as a hint for further node expansion. In the embodiment of the present application, the code generation model is a large language model, and sampling generation with a higher temperature is used to ensure the diversity of explored nodes.

[0092] In some embodiments, the trained code generation model can be used to assist in code development, for example:

[0093] During the code development phase, it provides code completion features, such as suggesting variable names, functions, syntax structures, etc.; it automatically detects potential errors in the code, such as suggesting or directly fixing common logic problems, syntax errors, or performance bottlenecks.

[0094] At the code testing node, code optimization suggestions are provided, such as improving algorithm efficiency, reducing redundant code, etc.; unit test code is generated to assist developers in verifying program behavior under different input conditions.

[0095] In some embodiments, when determining the evaluation values of different nodes in step 102, the following steps may also be performed:

[0096] Each node found in the tree search is evaluated based on a process reward model to obtain an evaluation value of each node.

[0097] Process Reward Models (PRMs) are an evaluation mechanism that assigns an evaluation value to each node in the tree search process, based on comprehensive considerations such as goal achievement, resource consumption, and uncertainty reduction. In some embodiments, the reward model can be a simple linear function or a complex nonlinear function. The specific setting is based on actual needs and is not limited in the embodiments of the present application.

[0098] In some embodiments, the next node to be executed is selected based on the evaluation values of different nodes. A greedy strategy can be used to select the node with the highest evaluation value, or an exploration-utilization strategy (such as the UCB1 algorithm) can be used to balance the exploration of unknown nodes and the utilization of known information. It should be noted that this narrative method is only an exemplary description and is not a specific limitation on the specific selection strategy. The embodiments of the present application do not limit the algorithm for selecting the next node.

[0099] In some embodiments, step 101 further includes:

[0100] Generate pseudocode based on the tree search thinking chain of the test case.

[0101] When generating pseudocode, first, clarify the basic process of tree search, including selecting nodes, expanding nodes, simulating and returning evaluation values. Secondly, describe how to select the next node to be processed from the current node. For example, compare the evaluation values or visit times of different nodes. Again, explain how to generate the child nodes of a node after selecting a node, that is, explore all possible actions of the node. Finally, explain how to simulate the results of the problem from the selected node until a terminal state is reached. It should be noted that this narrative method is only an exemplary description and is not a specific limitation on the specific pseudocode content. The embodiments of the present application do not limit this.

[0102] Generate the executable code for the test case based on the pseudocode, where the pseudocode is used to control the granularity of the tree search behavior.

[0103] Control the granularity of the inference steps according to the pseudocode, and convert each step in the pseudocode into specific programming language instructions to define the data structure.

[0104] When the executable code meets the test case, determine that the result reward value of the executable code is not zero.

[0105] During the process of tree search, when a leaf node is searched, it means that an executable code is generated. The search path from the root node to this leaf node is the generation path of this executable code. However, this executable code may not meet the test case, so it is marked with a result reward value. In practical applications, the specific value of the reward result value is determined according to the numerical range of the reward value. For example, if the range of the reward result value is 1 and 0, that is, when the executable code meets the test case, the reward result value is 1, and when it does not meet, the reward result value is 0.

[0106] When the generated executable code does not meet the requirements of the test case, it means that the generation path of this executable code is not the correct path, and the result reward value is set to 0; when the generated executable code can meet the requirements of the test case, it means that the generation path of this executable code is the correct path, and the result reward value is set to 1.

[0107] In some embodiments, the process of the code generation model executing the search tree according to the test case is the process of generating the executable code. In order to better control the granularity of the inference steps, the code generation model first generates a complete thought chain, and then generates the executable code step by step according to the thought chain. First, a detailed pseudocode is generated, and then the final executable code is generated under the guidance of this pseudocode. This method can not only adjust the detail level in the pseudocode to control the granularity of the inference behavior, but also generate different specific codes under the same pseudocode.

[0108] The pseudocode can better adjust and optimize the granularity of the inference steps in the search process, ensuring the accuracy and efficiency of the search. When the executable code meets the test case, the reward value is not zero, which helps to strengthen the positive feedback in the reinforcement learning process, thereby improving the learning effect and decision-making quality of the code generation model.

[0109] In some embodiments, the test case is generated by a result reward model; the evaluation of each node searched in the tree search based on the process reward model to obtain the evaluation value of each node further includes:

[0110] Calculate the weighted sum based on the process reward value of each node and the outcome reward value generated by the Outcome Reward Models (ORM), and calculate the evaluation value of each node.

[0111] Among them, when the node is a leaf node, it is determined that the weight coefficient of the outcome reward value is not zero.

[0112] In some embodiments, assign the outcome reward value of the executable code to the corresponding leaf node as the evaluation value of the leaf node. Through the evaluation value of the leaf node, it can be determined whether the search path from the leaf node to the root node is the correct path. Specifically, determine the relative importance of the process reward value and the outcome reward value in the total evaluation value by setting the weight coefficient. For each node, perform a weighted sum of the process reward value and the outcome reward value according to their respective weights. The calculation formula is:

[0113]

[0114] Among them, is the process reward value, is the outcome reward value, is the weight factor. When the node is a leaf node , otherwise .

[0115] That is, the evaluation value of the leaf node is the outcome reward value, and the evaluation value of other nodes is the process reward value. Take the calculated weighted sum as the evaluation value of the node and update the information of the corresponding node in the tree.

[0116] In some embodiments, when calculating the evaluation value, a time discount factor can also be considered to ensure the long-term consistency of the evaluation value.

[0117] By assigning a non-zero weight to the outcome reward value of the leaf node, the search algorithm is guided to preferentially select paths that can lead to good final results, enabling the algorithm to make more reasonable and effective decisions in a dynamic environment, thereby improving the efficiency and success rate of the overall decision-making system.

[0118] Optionally, after the tree search ends, the backward propagation update is performed according to the evaluation values of each node to update the state-action values and access counts of each node, including:

[0119] Perform a weighted average calculation on at least one evaluation value of each node to obtain the state-action value.

[0120] At the end of the i-th simulation, along the path from the leaf node Each edge (s, a) on the path to the root undergoes a backward pass update. The updates to its state-action value and visit count are performed according to the following rules:

[0121] ,

[0122] where is the node visit count, is the state-action value.

[0123] The denominator of the formula is , and the numerator is the sum of the evaluation values of all simulation results. The state-action value is calculated by taking a weighted average of the results of multiple simulations, where the weights are determined based on the proportion of each simulation result relative to the total number of simulations.

[0124] In some embodiments, after obtaining the correct search path in the tree search and storing the correct search path in the code generation model fine-tuning database as the training data for the code generation model, the method further includes:

[0125] Obtaining all search paths in the tree search and storing them in the process reward model training database in a preset format as the training data for the process reward model; wherein the preset format is test case, search path, and evaluation value of each node in the search path.

[0126] In some embodiments, pre-training the process reward model based on a pre-training dataset; wherein the pre-training dataset includes pre-training data of correct reasoning processes and correct answers, wrong reasoning processes and correct answers, and wrong reasoning processes and wrong answers.

[0127] After processing all problem-solving processes in the (where Q is the test case, s is the exploration path, and v is the evaluation value corresponding to this part of the reasoning process) format, it can be used as the training data for the process reward model to further train the process reward model; an initial value model is obtained through fine-tuning (SFT) on the base model using the training database to guide the LLM tree search for higher-quality solutions. The dataset includes various data such as correct reasoning process - correct answer, wrong reasoning process - correct answer, and wrong reasoning process - wrong answer.

[0128] The process reward model can determine the reward value based on the specific steps and intermediate results of the task, which makes the decision-making process of the model more interpretable. In the code generation task, the reward value can be used to determine why a certain path received a higher reward at a certain step, thereby improving the transparency and traceability of the model-generated code.

[0129] By obtaining all search paths of the tree search and storing them in a preset format, the training data of the process reward model is effectively accumulated, ensuring that the model can learn the evaluation values of different search paths, thereby improving the prediction accuracy and generalization ability of the model in complex decision-making scenarios, and providing data support for optimizing the search algorithm and improving the decision-making quality.

[0130] In some embodiments, through the self-training of the code generation model guided by the tree search, the code generation model can continuously self-correct according to the reward feedback in the tree search. When a potential valid code snippet is found, the reward value guidance of the process reward model can prompt the search to go deeper. When the search fails or an invalid code is generated, the code generation model will adjust the search strategy and generation method according to the feedback of the process reward, so as to perform self-optimization.

[0131] In some embodiments, before the code generation model performs a tree search based on the test case, the method further includes:

[0132] Performing preference training on the result reward model based on a preset preference dataset; wherein, the preset preference dataset includes positive examples and negative examples; the positive examples include training test cases that conform to the preferences of the preset preference dataset; the negative examples include training test cases that do not conform to the preferences of the preset preference dataset.

[0133] In some embodiments, the preset preference dataset further includes instructions, questions, and codes, as well as positive examples (test cases that conform to the preferences) and negative examples (test cases that are inconsistent with the preferences). The construction rules of the preference dataset are as follows: for positive examples, select three completely matching test cases; for negative examples, shuffle the outputs of three sampled test cases and concatenate the original inputs to generate input-output pairs that do not completely match, thereby constructing test cases that do not conform to the preferences.

[0134] By performing preference training on the result reward model based on the preset preference dataset, the model can learn to distinguish examples that conform to and do not conform to the preferences, improve the accuracy of preference decision-making in actual applications, and ensure that the output results are closer to the user's expectations.

[0135] In some embodiments, before performing preference training on the result reward model based on the preset preference dataset, the method further includes:

[0136] Performing supervised fine-tuning training on the result reward model according to the training dataset.

[0137] The core objective of supervised fine-tuning training is to ensure that the outputs generated by the result reward model follow a predetermined format, so that the generated test cases can be accurately parsed and extracted. The training data mainly comes from the TACO dataset. In some embodiments, the training dataset includes: questions, solutions, and test cases.

[0138] In some embodiments, the tree search algorithm is Monte Carlo Tree Search (MCTS), please refer to Figure 2 , Figure 2 which is a schematic flow diagram of a Monte Carlo tree search provided by an embodiment of the present application. As Figure 2 shown, the cylinder at the upper part of the figure represents a dataset, which contains a series of pairs, where is the test case question, is the cost of this question. Below is the decision sequence based on , to are decisions generated according to the policy under different conditions, and finally form a dataset containing ( , , , …, ).

[0139] Please continue to refer to Figure 2 , Figure 2 in which, ORM uses the information in the above dataset as input, PRM and Policy Model receive the dataset obtained from the left MCTS model as input, and the outputs of these modules are the right MCTS tree structure. The structure of the MCTS tree contains multiple nodes. The white nodes represent correct nodes, and the black nodes represent incorrect nodes. MCTS searches for the optimal decision path by continuously expanding and evaluating this tree structure.

[0140] The search framework based on MCTS has strong scalability in large-scale problems; as the task scale increases, the tree search can gradually expand the search space, and the process reward model ensures that the focus of the search is on the most likely successful path by providing guidance during the search process, thus avoiding inefficiencies caused by an overly large search space.

[0141] Corresponding to the above training method of the code generation model, the present invention also proposes a training device for the code generation model. Since the device embodiment of the present invention corresponds to the above method embodiment, details not disclosed in the device embodiment can be referred to the above method embodiment, and will not be elaborated in the present invention.

[0142] Figure 3 The structural schematic diagram of a training device for a code generation model provided by an embodiment of the present disclosure is as follows Figure 3 As shown, according to a second aspect of the present disclosure, there is provided a training device for a code generation model, including:

[0143] An execution unit 21, configured to obtain test cases and perform tree search based on the test cases;

[0144] An evaluation unit 22, configured to obtain evaluation values of each node in the tree search, and perform backward propagation update according to the evaluation values of each node to update the state-action values and access counts of each node;

[0145] A first acquisition unit 23, configured to obtain a correct search path in the tree search according to the state-action values and the access counts, and store the correct search path in a model fine-tuning database;

[0146] A first training unit 24, configured to train a code generation model based on the model fine-tuning database.

[0147] The training device for a code generation model provided by the present disclosure mainly includes the following technical solutions: obtaining test cases and performing tree search based on the test cases; obtaining evaluation values of each node in the tree search, and performing backward propagation update according to the evaluation values of each node to update the state-action values and access counts of each node; obtaining a correct search path in the tree search according to the state-action values and the access counts, and storing the correct search path in a model fine-tuning database; training a code generation model based on the model fine-tuning database. Compared with the related art, the embodiment of the present application combines tree search with a process reward model to ensure the pertinence and effectiveness of the search process. The node evaluation and backward propagation update mechanism enables the model to continuously optimize the state-action values and enhance the learning efficiency. By using high-quality samples generated by reward-guided tree search to self-train the code generation model, the ability of the model to generate complex codes is improved.

[0148] Further, in a possible implementation manner of the embodiment of the present disclosure, as shown Figure 4 The execution unit 21 is further configured to:

[0149] Evaluate each node searched in the tree search based on a process reward model to obtain the evaluation value of each node.

[0150] Further, in a possible implementation manner of the embodiment of the present disclosure, as shown Figure 4 The execution unit 21 includes:

[0151] The first generation module 211 is configured to generate pseudocode based on the tree search thought chain of the test case;

[0152] The second generation module 212 is configured to generate executable code for the test case based on the pseudocode; wherein, the pseudocode is used to control the tree search behavior granularity;

[0153] The determination module 213 is configured to determine that the result reward value of the executable code is not zero when the executable code meets the test case.

[0154] Further, in a possible implementation manner of the embodiment of the present disclosure, as Figure 4 shown, the test case is generated by a result reward model; the evaluation unit 22 is further configured to:

[0155] Calculate the weighted sum according to the process reward value and the result reward value of each node, and calculate the evaluation value of each node;

[0156] Wherein, when the node is a leaf node, it is determined that the weight coefficient of the result reward value is not zero.

[0157] Further, in a possible implementation manner of the embodiment of the present disclosure, as Figure 4 shown, the first acquisition unit 23 is further configured to:

[0158] Perform weighted average calculation on at least one evaluation value of each node respectively to obtain the state-action value.

[0159] Further, in a possible implementation manner of the embodiment of the present disclosure, as Figure 4 shown, the device further includes:

[0160] The second acquisition unit 25 is configured to obtain the correct search path in the tree search by the first training unit 24, store the correct search path in the code generation model fine-tuning database as the training data of the code generation model, and then obtain all search paths in the tree search and store them in the process reward model training database in a preset format as the training data of the process reward model; wherein, the preset format is the test case, the search path, and the evaluation value of each node in the search path.

[0161] Further, in a possible implementation manner of the embodiment of the present disclosure, as Figure 4 shown, the device further includes:

[0162] A second training unit 26, configured to perform preference training on the result reward model based on a preset preference data set before the execution unit 21 performs tree search based on the code generation model on the test case execution tree; wherein, the preset preference data set includes positive examples and negative examples; the positive examples include training test cases that conform to the preferences of the preset preference data set; the negative examples include training test cases that do not conform to the preferences of the preset preference data set.

[0163] Further, in a possible implementation manner of the embodiments of the present disclosure, the apparatus further includes:

[0164] The training unit is further configured to perform supervised fine-tuning training on the result reward model according to a training data set before performing preference training on the result reward model based on a preset preference data set.

[0165] Further, in a possible implementation manner of the embodiments of the present disclosure, the training data in the training data set includes questions, solutions, and test cases.

[0166] Further, in a possible implementation manner of the embodiments of the present disclosure, the preset preference data set further includes instructions, questions, and codes.

[0167] Further, in a possible implementation manner of the embodiments of the present disclosure, as Figure 4 shown, the apparatus further includes:

[0168] A third training unit 27, configured to perform pre-training on the process reward model based on a pre-training data set before the second acquisition unit 25 acquires all search paths in the tree search and stores them in the process reward model training database in a preset format as the training data of the process reward model; wherein, the pre-training data set includes pre-training data of correct reasoning processes and correct answers, incorrect reasoning processes and correct answers, and incorrect reasoning processes and incorrect answers.

[0169] It should be noted that the foregoing explanations of the method embodiments also apply to the apparatus of the embodiments of the present disclosure, with the same principle, and are not limited in the embodiments of the present disclosure.

[0170] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0171] Figure 5FIG. 0 shows a schematic block diagram of an exemplary electronic device 300 that may be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, for example, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as, personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0172] As Figure 5 shown, the device 300 includes a computing unit 301 that may perform various appropriate actions and processes in accordance with a computer program stored in a ROM (Read-Only Memory) 302 or a computer program loaded from a storage unit 308 into a RAM (Random Access Memory) 303. In the RAM 303, various programs and data required for the operation of the device 300 may also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O (Input / Output) interface 305 is also connected to the bus 304.

[0173] A plurality of components in the device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a magnetic disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the device 300 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0174] The computing unit 301 may be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any suitable processor, controller, microcontroller, etc. The computing unit 301 executes the various methods and processes described above, such as the training method of the code generation model. For example, in some embodiments, the training method of the code generation model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as the storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the methods described above may be executed. Alternatively, in other embodiments, the computing unit 301 may be configured to execute the aforementioned training method of the code generation model in any other suitable manner (e.g., by means of firmware).

[0175] The various embodiments of the systems and techniques described above in this document may be implemented in digital electronic circuitry, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chip), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0176] The program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0177] In the context of the present disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory), or a flash memory, an optical fiber, a CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0178] In order to provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or an LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0179] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0180] A computer system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"). The server can also be a server of a distributed system, or a server combined with blockchain.

[0181] Herein, it should be noted that artificial intelligence is a discipline that studies enabling a computer to simulate certain thinking processes and intelligent behaviors of humans (such as learning, reasoning, thinking, planning, etc.), and has both hardware-level technologies and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include several major directions such as computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, and knowledge graph technology.

[0182] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and no limitation is made herein.

[0183] The above specific embodiments do not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A training method for a code generation model, characterized in that, Including: Obtain test cases and perform tree search based on the test cases; Obtain the evaluation values of each node in the tree search, and perform backward propagation update according to the evaluation values of each node to update the state-action values and access counts of each node; Obtain the correct search path in the tree search according to the state-action values and the access counts, and store the correct search path in the model fine-tuning database; Train the code generation model based on the model fine-tuning database; Among them, the obtaining the evaluation values of each node in the tree search includes: Evaluate each node searched in the tree search based on the process reward model to obtain the evaluation values of each node; Among them, the test cases are generated by the result reward model; the evaluating each node searched in the tree search based on the process reward model to obtain the evaluation values of each node further includes: Calculate the weighted sum according to the process reward values and result reward values of each node to obtain the evaluation values of each node; among them, the evaluation value of a leaf node is the result reward value, the evaluation value of other nodes is the process reward value, and the weight coefficient of the result reward value of the leaf node is not zero; Among them, the obtaining test cases and performing tree search based on the test cases includes: Generate pseudocode based on the tree search thought chain of the test cases; Generate executable code for the test cases based on the pseudocode; where the pseudocode is used to control the behavior granularity of the tree search; When the executable code meets the test cases, determine that the result reward value of the executable code is not zero; When the executable code does not meet the test cases, determine that the result reward value of the executable code is zero.

2. The method according to claim 1, wherein The after the tree search ends, performing backward propagation update according to the evaluation values of each node to update the state-action values and access counts of each node includes: Perform weighted average calculation on at least one evaluation value of each node respectively to obtain the state-action value.

3. The method according to claim 1, wherein After obtaining the correct search path in the tree search and storing the correct search path in the code generation model fine-tuning database as the training data of the code generation model, the method further includes: Obtain all search paths in the tree search and store them in the process reward model training database in a preset format as the training data of the process reward model; where the preset format is test cases, search paths, and evaluation values of each node in the search path.

4. The method according to claim 1, wherein The code generation model performing tree search based on test cases further includes: Perform preference training on the result reward model based on a preset preference dataset; where the preset preference dataset contains positive examples and negative examples; the positive examples contain training test cases that conform to the preferences of the preset preference dataset; the negative examples contain training test cases that do not conform to the preferences of the preset preference dataset.

5. The method according to claim 4, characterized in that, Before performing preference training on the result reward model based on a preset preference dataset, the method further includes: Perform supervised fine-tuning training on the result reward model according to the training data set.

6. The method according to claim 5, wherein The training data in the training data set includes questions, solutions, and test cases.

7. The method according to claim 4, wherein The preset preference data set further includes instructions, questions, and code.

8. The method according to claim 3, wherein Before obtaining all search paths in the tree search and storing them in the process reward model training database in a preset format as the training data of the process reward model, the method further includes: Pre-train the process reward model based on the pre-training data set; wherein, the pre-training data set contains pre-training data of correct reasoning processes and correct answers, incorrect reasoning processes and correct answers, and incorrect reasoning processes and incorrect answers.

9. A training device for a code generation model, characterized in that, Includes: An execution unit for obtaining a test case and performing a tree search based on the test case; An evaluation unit for obtaining the evaluation values of each node in the tree search, and performing backward propagation update according to the evaluation values of each node to update the state-action values and access counts of each node; A first acquisition unit for obtaining the correct search path in the tree search according to the state-action value and the access count, and storing the correct search path in the model fine-tuning database; A training unit for training the code generation model based on the model fine-tuning database; Wherein, the execution unit is further configured to: Evaluate each node searched in the tree search based on the process reward model to obtain the evaluation value of each node; Wherein, the test case is generated by the result reward model; the evaluation unit is further configured to: Calculate the weighted sum according to the process reward value and the result reward value of each node, and calculate the evaluation value of each node; wherein, the evaluation value of the leaf node is the result reward value, and the evaluation value of other nodes is the process reward value, and the weight coefficient of the result reward value of the leaf node is not zero; Wherein, the execution unit includes: A first generation module for generating pseudo-code based on the tree search thought chain of the test case; A second generation module for generating executable code for the test case based on the pseudo-code; wherein, the pseudo-code is used to control the behavior granularity of the tree search; A determination module for determining that the result reward value of the executable code is not zero when the executable code meets the test case; The determination module is further configured to determine that the result reward value of the executable code is zero when the executable code does not meet the test case.

10. An electronic device, characterized in that, Includes: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

12. A computer program product, characterized in that Comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-8.

Images