Training methods and electronic devices for code generation models

Abstract

This application discloses a training method and electronic device for a code generation model, relating to the field of computer technology. The method includes: inputting sample prompts into a code generation model to obtain sample code; determining a first reward parameter to characterize execution stability based on the code execution information of the sample code, and parsing the code execution information, sample code, and sample prompts using a reference language model to obtain a second reward parameter to characterize the degree of syntax errors and a third reward parameter to characterize code style; adjusting the model parameters of the code generation model based on the first, second, and third reward parameters; and determining the target code generation model based on the adjusted code generation model when the adjusted model reaches convergence. This solves the technical problem of low training efficiency in code generation models and achieves the technical effect of improving the training efficiency of code generation models.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a code generation model training method and an electronic device. BACKGROUND

[0002] With the wide application of large language models in the field of program understanding and code generation, model training has become a key step to improve the performance of specific tasks. In related technologies, reinforcement learning with human feedback (RLHF) technology is usually used to train code generation models. However, the RLHF method needs a large amount of human participation to score the code segments generated by the model, which not only consumes a large amount of time and human resources, but also is difficult to scale, especially for complex or large-scale code generation tasks, the training efficiency is severely restricted.

[0003] In other words, the training method of the code generation model provided in the related technologies still has the technical problem of low training efficiency. SUMMARY

[0004] The present application provides a code generation model training method to at least solve the problem of low training efficiency of the code generation model in related technologies.

[0005] The present application provides a code generation model training method, comprising: inputting a sample prompt word into a code generation model to obtain a sample code matched with the sample prompt word; determining a first reward parameter for representing the execution stability of the sample code based on code running information corresponding to the sample code, and using a reference language model to parse the code running information, the sample code and the sample prompt word to obtain a second reward parameter for representing the syntax error degree of the sample code and a third reward parameter for representing the code style of the sample code; adjusting model parameters of the code generation model based on the first reward parameter, the second reward parameter and the third reward parameter to obtain an adjusted code generation model; and determining the adjusted code generation model as a target code generation model completed training in the case that the adjusted code generation model meets a convergence condition.

[0006] The application further provides a training device of a code generation model, comprising: a code generation unit configured to input a sample prompt word into the code generation model to obtain a sample code matched with the sample prompt word; a first determination unit configured to determine a first reward parameter used for representing execution stability of the sample code based on code running information corresponding to the sample code, and parse the code running information, the sample code and the sample prompt word by using a reference language model to obtain a second reward parameter used for representing a syntax error degree of the sample code and a third reward parameter used for representing a code style of the sample code; a model adjustment unit configured to adjust model parameters of the code generation model based on the first reward parameter, the second reward parameter and the third reward parameter to obtain an adjusted code generation model; and a second determination unit configured to determine the adjusted code generation model as a target code generation model completed with training in a case where the adjusted code generation model reaches a convergence condition.

[0007] The application further provides an electronic device, comprising: a memory configured to store a computer program; and a processor configured to implement steps of the training method of the code generation model when executing the computer program.

[0008] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is configured to implement steps of the training method of the code generation model when executed by a processor.

[0009] The application further provides a computer program product, comprising a computer program, and the computer program is configured to implement steps of the training method of the code generation model when executed by a processor.

[0010] In the embodiment of the application, the sample prompt word is input into the code generation model to obtain the sample code matched with the sample prompt word; the first reward parameter used for representing the execution stability of the sample code is determined based on the code running information corresponding to the sample code, and the code running information, the sample code and the sample prompt word are parsed by using the reference language model to obtain the second reward parameter used for representing the syntax error degree of the sample code and the third reward parameter used for representing the code style of the sample code; the model parameters of the code generation model are adjusted based on the first reward parameter, the second reward parameter and the third reward parameter to obtain the adjusted code generation model; and the adjusted code generation model is determined as the target code generation model completed with training in a case where the adjusted code generation model reaches the convergence condition. By adopting the embodiment of the application, the technical effect of improving the training efficiency of the code generation model is achieved, thereby solving the technical problem of low training efficiency of the code generation model. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0012] Figure 1 A flow chart of a code generation model training method provided by an embodiment of the present application is shown in FIG. 2.

[0013] Figure 2 A schematic diagram of a code generation model training method provided by an embodiment of the present application is shown in FIG. 3.

[0014] Figure 3 A schematic diagram of another code generation model training method provided by an embodiment of the present application is shown in FIG. 4.

[0015] Figure 4 A schematic diagram of a code generation model training system provided by an embodiment of the present application is shown in FIG. 5.

[0016] Figure 5 A schematic diagram of a code generation model training device provided by an embodiment of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present application.

[0018] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0019] In order to make those skilled in the art better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0020] As an optional solution, the code generation model training method described above, as shown in FIG. 2, comprises: Figure 1

[0021] ​S102, input the sample prompt word into the code generation model to obtain sample code matched with the sample prompt word.

[0022] It should be noted that the training method of the code generation model is applied to the training scene of the code generation model, specifically, but not limited to, at least applied to the following scenes:

[0023] 1) Automatic code repair and optimization. In the software development process, developers often need to repair code problems or optimize code performance. The large language model trained by the above-mentioned code generation model training method can automatically generate repair code according to the problem description or automatically generate an optimized version based on the given code.

[0024] 2) Code automatic generation: when developing new projects or functions, developers may need to write a large amount of code. The large language model trained by the above-mentioned code generation model training method can automatically generate code framework or completed code snippets according to project requirements or function descriptions.

[0025] 3) Intelligent code completion tool: in the development environment, the intelligent code completion tool can automatically complete the code snippet according to the input or context of the developer. The large language model trained by the above-mentioned code generation model training method can complete the code snippet.

[0026] Optionally, the sample prompt word is used to represent the natural language description of the programming task, such as task target, function requirement or description of code snippet, which aims to guide the code generation model to generate corresponding code. The sample code is used to indicate the code output by the code generation model based on the sample prompt word.

[0027] Further, the code generation model refers to a large language model that has been pre-trained and can generate corresponding code snippets according to the input natural language description (i.e. sample prompt word).

[0028] Optionally, inputting the sample prompt word into the code generation model to obtain the sample code matched with the sample prompt word refers to inputting the natural language description of the programming task into the code generation model, and the model generates corresponding code snippets based on the task description, aiming to realize the function of the task description.

[0029] S104, based on the code running information corresponding to the sample code, determine the first reward parameter for characterizing the execution stability of the sample code, and use the reference language model to analyze the code running information, the sample code and the sample prompt word, to obtain the second reward parameter for characterizing the syntax error degree of the sample code, and the third reward parameter for characterizing the code style of the sample code.

[0030] It should be noted that the first reward parameter includes a quantitative indicator for representing the execution stability of the sample code, which can be set high or low based on whether the running result of the code meets the expectation.

[0031] Optionally, the second reward parameter is used to quantify the degree of syntax error of the sample code, and the code running information is parsed by referring to the language model to identify the syntax error and give a corresponding score.

[0032] Further, the third reward parameter is used to evaluate the code style of the sample code, that is, the readability of the code and the programming specification followed, and is also obtained by referring to the parsing of the language model.

[0033] It should be noted that the first reward parameter for representing the execution stability of the sample code based on the code running information corresponding to the sample code means that after the code is generated, the code is executed and the running result (such as whether it passes the test, whether there is an exception, running time, etc.) is collected, and the execution stability of the code is determined according to these information, and the value of the first reward parameter is set accordingly.

[0034] Further, the second reward parameter for representing the degree of syntax error of the sample code and the third reward parameter for representing the code style of the sample code obtained by parsing the code running information, the sample code and the sample prompt word by using the reference language model can be but not limited to realized by the following way: using another pre-trained language model, such as a more powerful teacher model, to analyze and evaluate the generated code fragment, and calculating the second reward parameter and the third reward parameter respectively according to the syntax structure, error type, code style and other characteristics, to quantify the syntax quality and programming style of the code.

[0035] S106, based on the first reward parameter, the second reward parameter and the third reward parameter, adjusting the model parameters of the code generation model to obtain an adjusted code generation model.

[0036] Optionally, the model parameters refer to the parameters such as weights and biases inside the code generation model, which determine the generation ability and behavior of the model.

[0037] It should be noted that the adjustment of the model parameters of the code generation model based on the first reward parameter, the second reward parameter and the third reward parameter to obtain the adjusted code generation model means that the three reward parameters are fused, the target reward function is calculated by the fused reward parameters, and then the parameters of the code generation model are updated based on the target reward function to optimize the generation ability of the model.

[0038] S108, in the case that the adjusted code generation model reaches the convergence condition, the adjusted code generation model is determined as the target code generation model completed training.

[0039] It should be noted that the convergence condition includes but is not limited to the model performance index reaching or exceeding the preset threshold, the training loss no longer significantly decreasing within several rounds, etc., which is an important basis for judging whether the model training is completed.

[0040] Optionally, the case that the adjusted code generation model reaches the convergence condition can be but is not limited to indicating that, in the model training process, once the adjustment of the model parameters no longer significantly improves the quality of code generation or the comprehensive performance index of the model reaches the preset standard, it can be considered that the model training has been completed, and the code generation model at this time is the target code generation model, which can be used for subsequent code generation tasks.

[0041] In the embodiment of the present application, the sample prompt word is input into the code generation model to obtain sample code matched with the sample prompt word; based on the code running information corresponding to the sample code, a first reward parameter for representing the execution stability of the sample code is determined, and the code running information, the sample code and the sample prompt word are parsed by using the reference language model to obtain a second reward parameter for representing the syntax error degree of the sample code and a third reward parameter for representing the code style of the sample code; based on the first reward parameter, the second reward parameter and the third reward parameter, the model parameters of the code generation model are adjusted to obtain an adjusted code generation model; in the case that the adjusted code generation model reaches the convergence condition, the adjusted code generation model is determined as a target code generation model that has completed training. By using the embodiment of the present application, the model is guided by the sample text to generate code segments related to the task, ensuring the pertinence and task relevance of the generated code and improving the practicability of the model. Using the feedback information after code execution as a direct basis for model training can objectively evaluate the execution effect of the code, which is more efficient and scalable than manual scoring. By using another trained model to evaluate syntax errors and code style, not only can an automatic and objective evaluation standard be provided, but also the process can be completed without the direct participation of humans, greatly saving human resources. In summary, by using the embodiment of the present application, the automation and efficiency of code generation model training are realized, especially in evaluating the execution stability, syntax correctness and style quality of the code, the joint use of running information and reference model ensures the comprehensiveness and objectivity of the feedback, greatly improving the model training efficiency and code generation quality, thereby solving the technical problem of low training efficiency of the code generation model.

[0042] As an optional solution, determining the first reward parameter for representing the execution stability of the sample code based on the running result of the sample code includes:

[0043] S1, obtaining a running result of the sample code from the code running information.

[0044] S2, determining the first parameter value of the first reward parameter as a first numerical value in a case where the running result indicates that the sample code runs successfully.

[0045] S3, determining the first parameter value as a second numerical value in a case where the running result indicates that the sample code runs unsuccessfully, wherein the first numerical value is greater than the second numerical value.

[0046] It should be noted that the sample code refers to a piece of code generated by the code generation model according to a given sample prompt word, which is used to reflect the generation ability and accuracy of the model in a specific programming task.

[0047] Optionally, the code running information can include but is not limited to the detailed status of code execution, such as standard output, exception information, backtracking information, and running result and running time, etc., for comprehensive evaluation of the execution behavior of the code.

[0048] It should be noted that the running result is feedback information about whether the code function is correct and whether it achieves the expected purpose after the code execution is completed, including pass or fail status.

[0049] Optionally, as an optional example, the above content can be exemplarily explained and described by the following example: for each generated code, the functional correctness reward reward1 (i.e., the first reward parameter) is determined according to the flag indicating whether the test case is passed, if the flag indicates that the running is successful, the correctness reward is determined as 1, otherwise as 0.

[0050] In the embodiment of the present application, the running result of the sample code is obtained from the code running information; in a case where the running result indicates that the sample code runs successfully, the first parameter value of the first reward parameter is determined as a first numerical value; in a case where the running result indicates that the sample code runs unsuccessfully, the first parameter value is determined as a second numerical value, wherein the first numerical value is greater than the second numerical value. By directly converting the running result into the first numerical value and the second numerical value of the reward parameter, the model can quickly learn from the feedback, reward or punish the execution stability of the code, speed up the learning curve in the model training process, and improve the training efficiency.

[0051] As an optional solution, after inputting the sample prompt word into the code generation model to obtain the sample code matched with the sample prompt word, the method further comprises:

[0052] integrating the sample prompt word, the code running information and the sample code to obtain target structure data, wherein the target structure data is used to store the association relationship between the sample prompt word, the code running information and the sample code.

[0053] Optionally, the aforementioned target structured data is a type of structured data that combines sample prompts, code execution information, and sample code. It is used to establish relationships between these elements, enabling the model to easily obtain comprehensive feedback information during training. For example, the aforementioned target structured data can, but is not limited to, refer to... Figure 2 The data format shown.

[0054] In this embodiment, sample prompts, code execution information, and sample code are integrated to obtain target structure data. This target structure data stores the relationships between sample prompts, code execution information, and sample code. By employing this embodiment, the establishment of target structure data enables efficient association and retrieval of various parts involved in model training (prompts, generated results, and execution feedback), improving the organization of the training process and data utilization efficiency, thereby optimizing the overall training effect.

[0055] As an optional approach, a reference language model is used to parse code execution information, sample code, and sample prompts to obtain a second reward parameter characterizing the degree of syntax errors in the sample code, and a third reward parameter characterizing the code style of the sample code, including:

[0056] The target structure data and the first prompt word used to guide the reference language model in generating the second reward parameter are input into the reference language model so that the reference language model can parse the target structure data and obtain the second parameter value corresponding to the second reward parameter. The second parameter value is negatively correlated with the degree of syntax error in the sample code.

[0057] It should be noted that the first prompt word mentioned above is a specially designed natural language to guide the reference language model in evaluating syntax errors. It may contain instructions such as "detect syntax errors" or "analyze code syntax".

[0058] It should be noted that the value of the second parameter mentioned above can be achieved, but is not limited to, by referencing the output probability or score of the language model. It is negatively correlated with the degree of grammatical error, that is, the more grammatical error there is, the lower the parameter value.

[0059] Optionally, as an example, the above content can be illustrated by, but is not limited to, the following example: The output of the model is input into the teacher model. For serious runtime errors (such as infinite loops, memory overflows, etc.) as judged by the test results or the teacher model, a penalty term reward2 (used to represent the second reward parameter mentioned above) is introduced. For example, the teacher model is input: Please evaluate the runtime stability of the following code and judge serious runtime errors. The score range is -1 to 0, with 0 being the most stable.

[0060] The target structure data and the second cue word used to guide the reference language model in generating the third reward parameter are input into the reference language model so that the reference language model can parse the target structure data and obtain the third parameter value corresponding to the third reward parameter. The third parameter value is positively correlated with the code style quality of the sample code.

[0061] It should be noted that the second prompt mentioned above is used to guide the reference language model in evaluating code style, and may include instructions such as code indirection, code readability, and code standardization, to ensure the accuracy of code style evaluation.

[0062] Optionally, the third reward parameter mentioned above is used to characterize the code style quality of the sample code. The higher the value, the more the code style conforms to the specifications and standards, and the higher the quality of the code generated by the model.

[0063] Optionally, as an example, the above content can be illustrated by, but is not limited to, the following example: Input the output of the model into the teacher model and give an additional reward of reward3 (used to represent the third reward parameter mentioned above) for the style of the code. For example, input the following prompt into the teacher model: Please evaluate the conciseness and readability of the following code and give a score ranging from -1 to 0, where 0 represents the best code style.

[0064] In this embodiment, target structural data and a first prompt word used to guide the reference language model in generating a second reward parameter are input into the reference language model. The reference language model then parses the target structural data to obtain a second parameter value corresponding to the second reward parameter. The second parameter value is negatively correlated with the degree of syntactic errors in the sample code. Similarly, target structural data and a second prompt word used to guide the reference language model in generating a third reward parameter are input into the reference language model. The reference language model then parses the target structural data to obtain a third parameter value corresponding to the third reward parameter. The third parameter value is positively correlated with the code style quality of the sample code. By employing this embodiment, combined with the intelligent evaluation of the reference model, a balanced improvement in both syntactic correctness and code style optimization can be ensured in the code generation model. This results in generated code that is not only highly functional but also improves readability and maintainability.

[0065] As an optional approach, the model parameters of the code generation model are adjusted based on the first reward parameter, the second reward parameter, and the third reward parameter, resulting in an adjusted code generation model including:

[0066] S1, perform weighted fusion processing on the first reward parameter, the second reward parameter, and the third reward parameter to obtain the target reward parameter.

[0067] In this embodiment, asFigure 3 As shown, the first reward parameter used to characterize the sample function accuracy reward, the second reward parameter used to penalize the sample code running stability, and the third reward parameter used to incentivize the code structure simplicity are weighted and summed based on different weights (i.e., W1, W2, W3) to obtain the target reward parameter.

[0068] S2, adjust the model parameters of the code generation model using the target reward parameters to obtain the adjusted code generation model.

[0069] Optionally, the above-mentioned adjustment of the model parameters of the code generation model using the target reward parameter to obtain the adjusted code generation model may include, but is not limited to: generating a target loss value corresponding to the code generation model based on the target reward parameter, sample prompt words, and sample code; and adjusting the model parameters of the code generation model using the target loss value to obtain the adjusted code generation model.

[0070] In this embodiment, the first reward parameter, the second reward parameter, and the third reward parameter are weighted and fused to obtain the target reward parameter. The model parameters of the code generation model are then adjusted using the target reward parameter to obtain the adjusted code generation model. By using this embodiment, the model parameter adjustment is guided by the target reward parameter, which can more accurately guide the model to learn the optimization strategy for code generation, accelerate the model's convergence process, and improve training efficiency and the quality of the final code generation.

[0071] As an alternative approach, the model parameters of the code generation model can be adjusted using the target reward parameter, resulting in an adjusted code generation model including:

[0072] Based on the target reward parameters, sample prompts, and sample codes, the target loss value corresponding to the code generation model is generated.

[0073] It should be noted that the target loss value mentioned above can be calculated, but is not limited to, based on the OddsRatio Preference Optimization (ORPO) algorithm.

[0074] For example, the above content can be illustrated using examples, but is not limited to the following:

[0075] L(θ)=E(x,a,r)∼D[r⋅logπθ(a∣x)]−λ⋅KL(πθ∥πβ)(1)

[0076] Where L(θ) represents the target loss value.

[0077] Furthermore, E(x, a, r)∼D is used to represent the expected value, that is, the expected value for all combinations of (x, a, r) ​​under the data distribution D (e.g., a training sample set consisting of multiple sample codes obtained from multiple sample parameters). Here, x is the input sample cue word, a is the action generated by the model (i.e., the output code), and r is the reward obtained based on the code generated by the model and the test results (i.e., the target reward parameter).

[0078] Optionally, the formula r⋅logπθ(a|x) represents the policy optimization objective in reinforcement learning. logπθ(a|x) is the log probability of the current model (i.e., the code generation model) generating an action given the input. By maximizing the expected value of r⋅logπθ(a|x), the model is guided to increase the probability of generating actions (code) that yield higher rewards, thereby improving the quality and execution accuracy of the output code.

[0079] Furthermore, λ⋅KL(πθ∥πβ) is the KL divergence penalty term, where λ is an adjustment parameter controlling the degree of influence of the penalty term. KL(πθ∥πβ) represents the KL divergence between the current model (πθ) and the reference model (πβ), used to measure the difference between the two probability distributions. The KL divergence penalty term prevents the model policy from deviating excessively from the reference model during the optimization process, thereby avoiding the risk of overfitting or generating degenerate samples when the model pursues a specific reward structure, and helping to maintain the stability of the model.

[0080] In other words, this embodiment combines the policy optimization objective and the KL divergence penalty mechanism from reinforcement learning to guide the update of parameters in a large language model. Specifically, it aims to maximize the expected reward of the code generated by the model, while maintaining the continuity and stability of the model policy by limiting the difference between the model and a predefined reference model. This prevents the model from aggressively changing its policy during training, which could lead to performance degradation or the generation of uncontrollable code.

[0081] The model parameters of the code generation model are adjusted using the target loss value to obtain the adjusted code generation model.

[0082] It should be noted that the above-described method of adjusting the model parameters of the code generation model using the target loss value to obtain the adjusted code generation model can be used, but is not limited to, to indicate the process by which the model updates its internal parameters using the target loss value in the backpropagation algorithm. Specifically, the model calculates the gradient of the loss value with respect to the model parameters and uses the optimizer to adjust the model parameters according to the principle of gradient descent, thereby obtaining a version of the code generation model with updated parameters.

[0083] In this embodiment, a target loss value is generated for the code generation model based on the target reward parameter, sample prompts, and sample code. The model parameters of the code generation model are then adjusted using the target loss value to obtain the adjusted code generation model. This embodiment ensures the directionality and effectiveness of the model parameter adjustment. Through backpropagation of the target loss value, deviations in code generation can be quickly corrected, driving the model to evolve towards higher-quality code generation.

[0084] As an optional approach, before adjusting the model parameters of the code generation model based on the first reward parameter, the second reward parameter, and the third reward parameter to obtain the adjusted code generation model, the following steps are also included:

[0085] S1, after performing the i-th round of training on the code generation model, obtain the N target reward parameters generated during the i-th round of training on the code generation model, and the N target reward parameters generated during the (i-1)-th round of training on the code generation model, where i is a positive integer greater than 2 and N is a positive integer greater than 1.

[0086] S2, if the difference between the average parameter value of the N target reward parameters corresponding to the i-th round of training and the average parameter value of the N target reward parameters corresponding to the i-1th round of training is less than a predetermined threshold, it is determined that the code generation model that has completed the i-th round of training has reached the convergence condition.

[0087] S3 determines the target code generation model from the code generation model that has completed the i-th round of training.

[0088] Optionally, the i mentioned above refers to the number of training epochs specified during the training process. It is a positive integer greater than 2, used to mark the stage progress of model training.

[0089] It should be noted that N above is a positive integer, representing the number of reward parameters collected during each training process, used to evaluate multiple dimensions or samples of model performance.

[0090] Optionally, the above N rounds of training refers to the number of times the code generation model is trained iteratively. Here, N is a positive integer greater than 1, representing the number of loops or iterations during the training process.

[0091] It should be noted that the above M target reward parameters are the set of reward parameters produced by the model during N rounds of training. M is greater than N, which means that in each round of training, the model may generate multiple code snippets, each of which has its corresponding reward parameters.

[0092] Furthermore, the aforementioned average parameter value refers to the mean obtained by statistical analysis of the M target reward parameters, which is one of the important indicators for evaluating the overall training effect of the model.

[0093] Optionally, the aforementioned predetermined parameter values ​​are used to determine whether the model has achieved the expected training effect, and are usually related to the ideal performance of the model or the expected average value of the target reward parameter.

[0094] Optionally, but not limited to, the above content can be illustrated with examples:

[0095] Regarding training epoch control: training is iterated in units of training epochs, with 10,000 samples per epoch. The recommended default number of iterations is 5. A gradient optimizer is used for efficient fine-tuning, and the model is backpropagated based on the reward function output.

[0096] For model evaluation and updates: After each training round, 200 new tasks can be randomly sampled for generation and testing. The performance of the model before training and the current model is compared in terms of pass rate, stability, execution efficiency, etc. If the average improvement of the target reward parameter no longer exceeds a threshold (e.g., 0.05), further training is terminated to prevent overfitting.

[0097] In this embodiment, after performing the i-th round of training on the code generation model, N target reward parameters generated during the i-th round of training and N target reward parameters generated during the (i-1)-th round of training are obtained, where i is a positive integer greater than 2 and N is a positive integer greater than 1. If the difference between the average parameter value of the N target reward parameters corresponding to the i-th round of training and the average parameter value of the N target reward parameters corresponding to the (i-1)-th round of training is less than a predetermined threshold, the code generation model that has completed the i-th round of training is determined to have reached the convergence condition. The code generation model that has completed the i-th round of training is then used to determine the target code generation model. This embodiment optimizes the training process, avoids resource waste, and ensures that the model can terminate training in a timely manner after reaching a certain performance level, thereby improving the efficiency and effectiveness of model training.

[0098] As an optional approach, after inputting the sample prompt words into the code generation model to obtain sample codes that match the sample prompt words, the following steps are also included:

[0099] S1, obtain code test cases that match the sample code.

[0100] S2 runs sample code using code test cases.

[0101] S3 retrieves the output information, exception information, and backtracking information generated during the execution of the sample code.

[0102] S4: When the sample code finishes running, obtain the execution result and runtime of the sample code.

[0103] The code execution information includes output information, exception information, backtracking information, execution results, and runtime.

[0104] Optionally, the above code test cases refer to a set of input data or a preset scenario used to test the correctness and logical function of a specific code snippet, with the purpose of verifying whether the code can run correctly as expected.

[0105] Furthermore, the above output information refers to the standard output or return value generated by the program or script after the code snippet is executed, which is used to evaluate the functionality of the code.

[0106] It should be noted that the above-mentioned exception information includes any unexpected runtime errors or interruptions that may occur during the execution of the code snippet, and is an important basis for judging the robustness of the code.

[0107] Optionally, the aforementioned backtracking information refers to the error stack trace information generated by the system when an exception is encountered during code execution. It details the location and possible cause of the exception.

[0108] Furthermore, the above results include the final conclusion of whether the code has passed all test cases, which is an indicator that directly reflects the correctness of the code's functionality.

[0109] It should be noted that the runtime mentioned above is the time taken from the start to the end of the code snippet execution, and is used to evaluate the execution efficiency of the code.

[0110] In this embodiment, code test cases matching the sample code are obtained; the sample code is run using the code test cases; output information, exception information, and backtracking information generated by the sample code during execution are obtained; and the execution result and runtime of the sample code are obtained when the sample code finishes running. The code execution information includes output information, exception information, backtracking information, execution result, and runtime. By using this embodiment, the model can gain a more comprehensive understanding of the shortcomings in code generation through feedback obtained from running test cases, thereby enabling targeted parameter adjustments in subsequent training to improve the stability and correctness of code generation.

[0111] As an optional approach, obtaining code test cases that match the sample code includes:

[0112] S1. Input the third prompt word and sample code into the reference language model. The third prompt word is used to guide the reference language model to generate code test cases.

[0113] S2 uses a reference language model to parse the sample code and generate code test cases.

[0114] In this embodiment of the application, a third prompt word and sample code are input into the reference language model, wherein the third prompt word is used to guide the reference language model to generate code test cases;

[0115] The sample code is parsed using a reference language model to generate code test cases.

[0116] Optionally, the aforementioned third prompt word is used to guide the reference language model in constructing specific code test cases, which are designed to verify the functional correctness and stability of the sample code.

[0117] Optionally, as an example, the above can be explained by, but is not limited to, the following example: using a more capable teacher model to generate test scripts corresponding to the code based on the semantics of the third cue word and each code segment.

[0118] By employing the embodiments of this application, test cases are generated based on the third prompt word using a reference language model, which can intelligently provide targeted testing for the code generated by the model, further improving the automation and efficiency of training.

[0119] As an alternative approach, the training methods for the above code generation models also include:

[0120] Each time sample code is generated, the weights of the first, second, and third reward parameters are dynamically adjusted based on the complexity of the sample code. The code complexity is determined by indicators such as the number of lines of code, the number of function call levels, or the number of nested loops, which are used to reflect the logical complexity and execution difficulty of the code.

[0121] Optionally, the code complexity mentioned above refers to a quantitative measurement of the complexity of code structure and logic, which may include the number of lines of code, the number of function call levels, the number of nested loop levels, etc. It is used to reflect the effort required to implement the code's functionality and the potential error rate.

[0122] Furthermore, the aforementioned dynamic weight adjustment mechanism refers to automatically adjusting the weights of the first, second, and third reward parameters based on the complexity of each generated sample code, in order to better balance the execution stability, syntax correctness, and code style of the code.

[0123] Furthermore, the aforementioned dynamic adjustment of the weights of the first, second, and third reward parameters based on the complexity of the sample code each time it is generated can be used, but is not limited to, to indicate that after each code snippet is generated by the code generation model, the weights of the reward parameters are adjusted based on the complexity metrics of that code snippet (such as the number of lines of code, the number of function call layers, or the number of nested loop layers) to ensure that more emphasis is placed on the execution stability of highly complex code, while strengthening the constraints on style and syntax for simple code.

[0124] The embodiments of this application introduce a technical problem that helps to balance the logical complexity of the code with its implementation efficiency and maintainability, and avoids the model from focusing too much on a certain reward parameter, resulting in the generated code that performs well in some aspects but has significant shortcomings in others.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0126] As an optional embodiment, it can be based on, but is not limited to, such as Figure 4 The training system for the code generation model shown is used to implement the training method of the above-mentioned code generation model. The system includes:

[0127] Code sample generation module 402: Receives instructions or programming task text described by the user in natural language, passed in as a prompt; the large language model to be trained generates code based on the passed prompt. It supports various sampling strategies, such as Top-k Selection (Top-k), Nucleus Sampling (Top-p), or temperature, to generate multiple candidate codes.

[0128] Test Agent Module 404: Automatically builds or retrieves relevant test cases for the current task using the teacher's large language model, and executes the tests;

[0129] Reward function construction module 406: Calculates multi-dimensional reward values ​​based on test results;

[0130] Large Model Fine-Tuning Module 408: Employs the ORPO algorithm, uses a reference model to adjust strategies and update parameters; manages training rounds, log recording, and performance monitoring.

[0131] Embodiments of this application also provide a training apparatus for a code generation model, such as... Figure 5 As shown, the device includes:

[0132] The code generation unit 502 is used to input sample prompt words into the code generation model to obtain sample codes that match the sample prompt words;

[0133] The first determining unit 504 is used to determine a first reward parameter for characterizing the execution stability of the sample code based on the code execution information corresponding to the sample code, and to parse the code execution information, sample code and sample prompt words using a reference language model to obtain a second reward parameter for characterizing the degree of syntax error of the sample code and a third reward parameter for characterizing the code style of the sample code.

[0134] The model adjustment unit 506 is used to adjust the model parameters of the code generation model based on the first reward parameter, the second reward parameter, and the third reward parameter to obtain the adjusted code generation model.

[0135] The second determining unit 508 is used to determine the adjusted code generation model as the target code generation model for completing training when the adjusted code generation model reaches the convergence condition.

[0136] Optionally, in this embodiment, the first determining unit includes: an acquisition module, configured to acquire the execution result of the sample code from the code execution information; a determining module, configured to determine the first parameter value of the first reward parameter as a first value when the execution result indicates that the sample code has executed successfully; and a first determining module, configured to determine the first parameter value as a second value when the execution result indicates that the sample code has failed to execute, wherein the first value is greater than the second value.

[0137] Optionally, in this embodiment, the above-mentioned device further includes: a parameter integration unit, used to integrate sample prompt words, code execution information and sample code to obtain target structure data, wherein the target structure data is used to store the correlation between sample prompt words, code execution information and sample code.

[0138] Optionally, in this embodiment, the first determining unit further includes: a second determining module, used to input the target structure data and a first prompt word used to guide the reference language model to generate a second reward parameter into the reference language model, so that the reference language model parses the target structure data to obtain a second parameter value corresponding to the second reward parameter, wherein the second parameter value is negatively correlated with the degree of syntax error of the sample code; and a third determining module, used to input the target structure data and a second prompt word used to guide the reference language model to generate a third reward parameter into the reference language model, so that the reference language model parses the target structure data to obtain a third parameter value corresponding to the third reward parameter, wherein the third parameter value is positively correlated with the code style quality of the sample code.

[0139] Optionally, in this embodiment, the model adjustment unit includes: a fusion processing module, used to perform weighted fusion processing on the first reward parameter, the second reward parameter, and the third reward parameter to obtain the target reward parameter; and an adjustment module, used to adjust the model parameters of the code generation model using the target reward parameter to obtain the adjusted code generation model.

[0140] Optionally, in this embodiment, the adjustment module is further configured to: generate a target loss value corresponding to the code generation model based on the target reward parameter, sample prompt words, and sample code; and adjust the model parameters of the code generation model using the target loss value to obtain the adjusted code generation model.

[0141] Optionally, in this embodiment, the above-mentioned apparatus further includes: an acquisition unit, configured to acquire N target reward parameters generated during the i-th round of training of the code generation model, and N target reward parameters generated during the (i-1)-th round of training of the code generation model, wherein i is a positive integer greater than 2 and N is a positive integer greater than 1; a third determination unit, configured to determine that the code generation model that has completed the i-th round of training has reached the convergence condition if the difference between the average parameter value of the N target reward parameters corresponding to the i-th round of training and the average parameter value of the N target reward parameters corresponding to the (i-1)-th round of training is less than a predetermined threshold; and a fourth determination unit, configured to determine the code generation model that has completed the i-th round of training as the target code generation model.

[0142] Optionally, in this embodiment, the above-mentioned device further includes: a first acquisition unit, used to acquire code test cases that match the sample code; a code execution unit, used to run the sample code using the code test cases; a second acquisition unit, used to acquire output information, exception information, and backtracking information generated by the sample code during execution; and a third acquisition unit, used to acquire the execution result and runtime of the sample code when the sample code finishes execution; wherein the code execution information includes output information, exception information, backtracking information, execution result, and runtime.

[0143] Optionally, in this embodiment, the first acquisition unit is further configured to: input the third prompt word and the sample code into the reference language model, wherein the third prompt word is used to guide the reference language model to generate code test cases; and use the reference language model to parse the sample code to generate code test cases.

[0144] For a description of the features in the embodiment corresponding to the training device of the code generation model, please refer to the relevant description of the embodiment corresponding to the training method of the code generation model, which will not be repeated here.

[0145] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described code generation model training method embodiments.

[0146] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the training method embodiments of any of the above-described code generation models at runtime.

[0147] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0148] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the training method embodiments of any of the above-described code generation models.

[0149] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the training method embodiments of any of the above-described code generation models.

[0150] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0151] The training method for a code generation model provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for training a code generation model, characterized in that, The method comprises the following steps: inputting a sample prompt word into a code generation model to obtain a sample code matched with the sample prompt word; determining a first reward parameter for representing execution stability of the sample code based on code running information corresponding to the sample code; inputting target structure data and a first prompt word into a reference language model to obtain a second reward parameter for representing a syntax error degree of the sample code, wherein the target structure data is used to store an association relationship among the sample prompt word, the code running information and the sample code, and the first prompt word is used to guide the reference language model to generate the second reward parameter; inputting the target structure data and a second prompt word into the reference language model to obtain a third reward parameter for representing a code style of the sample code, wherein the second prompt word is used to guide the reference language model to generate the third reward parameter; determining respective weights of the first reward parameter, the second reward parameter and the third reward parameter according to a complexity of the sample code; performing weighted fusion processing on the first reward parameter, the second reward parameter and the third reward parameter based on the respective weights of the first reward parameter, the second reward parameter and the third reward parameter to obtain a target reward parameter; adjusting model parameters of the code generation model by using the target reward parameter to obtain an adjusted code generation model; in a case where the adjusted code generation model meets a convergence condition, determining the adjusted code generation model as a target code generation model completed with training.

2. The method of claim 1, wherein, The method further comprises the following steps after the step of inputting the sample prompt word into the code generation model to obtain the sample code matched with the sample prompt word: integrating the sample prompt word, the code running information and the sample code to obtain the target structure data.

4. The method of claim 3, wherein: the step of inputting the target structure data and the first prompt word into the reference language model to obtain the second reward parameter for representing the syntax error degree of the sample code comprises: inputting the target structure data and the first prompt word into the reference language model, so that the reference language model analyzes the target structure data to obtain a second parameter value corresponding to the second reward parameter, wherein the second parameter value is negatively correlated with the syntax error degree of the sample code.

3. The method of claim 1, wherein, ​ ​ ​ ​ The inputting the target structure data and the second prompt word into the reference language model to obtain a third reward parameter for representing a code style of the sample code comprises: inputting the target structure data and the second prompt word into the reference language model, so that the reference language model parses the target structure data to obtain a third parameter value corresponding to the third reward parameter, wherein the third parameter value is positively correlated with the code style quality of the sample code.

5. The method of claim 1, wherein, The adjusting the model parameter of the code generation model by using the target reward parameter to obtain the adjusted code generation model comprises: generating a target loss value corresponding to the code generation model based on the target reward parameter, the sample prompt word and the sample code; adjusting the model parameter of the code generation model through the target loss value to obtain the adjusted code generation model.

6. The method of Claim 5, wherein, Before the adjusting the model parameter of the code generation model by using the target reward parameter to obtain the adjusted code generation model, the method further comprises: after the i-th round of training of the code generation model is performed, obtaining N target reward parameters generated in the i-th round of training of the code generation model and N target reward parameters generated in the i-1-th round of training of the code generation model, wherein i is a positive integer greater than 2, and N is a positive integer greater than 1; in a case where a difference between an average parameter value of the N target reward parameters corresponding to the i-th round of training and an average parameter value of the N target reward parameters corresponding to the i-1-th round of training is less than a predetermined threshold, determining that the code generation model after the i-th round of training reaches a convergence condition; determining the code generation model after the i-th round of training as the target code generation model. 7.The method of Claim 1 to 6, wherein, After the inputting the sample prompt word into the code generation model to obtain the sample code matched with the sample prompt word, the method further comprises: obtaining a code test case matched with the sample code; running the sample code by using the code test case; obtaining output information, exception information and backtracking information generated in a running process of the sample code; in a case where the running of the sample code is ended, obtaining a running result of the sample code and a running duration of the sample code; wherein the code running information comprises the output information, the exception information, the backtracking information, the running result and the running duration.

8. The method of Claim 7, wherein, The obtaining the code test case matched with the sample code comprises: inputting a third prompt word and the sample code into the reference language model, wherein the third prompt word is used to guide the reference language model to generate the code test case; parsing the sample code by using the reference language model to generate the code test case.

9. An electronic device, comprising: comprise: a memory for storing a computer program; a processor for implementing the steps of the training method of the code generation model according to any one of claims 1 to 8 when executing the computer program.

Images