Code generation

Abstract

Embodiments of the present disclosure provide a scheme for code generation. A method includes receiving a query indicative of a code generation problem; generating, using a target machine learning model, a first code segment based on the query and a prefix code, wherein the target machine learning model is constructed based on a language model and configured to iteratively generate code segments to form computer code for the code generation problem; determining, using a trained reward model, a first reward score corresponding to the first code segment based on an updated prefix code, the first reward score indicative of correctness of the first code segment with respect to the code generation problem, wherein the first code segment is appended to the prefix code to obtain the updated prefix code; and fine-tuning the target machine learning model based on at least the first reward score through reinforcement learning.

Description

Technical Field

[0001] The disclosed example embodiments generally relate to machine learning, and more specifically to methods, apparatus, devices, and computer-readable storage media for code generation. Background Technology

[0002] Rapid advancements in language models (LMs) have revolutionized code generation, enabling models to achieve near-human performance on programming tasks. These models have demonstrated the ability to generate syntactically correct and functionally feasible code snippets, significantly aiding the software development process. Building on these successes, recent research has explored using reinforcement learning (RL) based on unit test feedback to further enhance the code generation capabilities of LMs. By incorporating unit tests as a reward mechanism, these methods aim to guide LM-generated code not only to compile but also to pass specific test cases, thereby improving the overall reliability and quality of the code. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for code generation is provided. The method includes: receiving a query indicating a code generation problem; generating a first code segment based on the query and prefix code using a target machine learning model, wherein the target machine learning model is built based on a language model and configured to iteratively generate code segments to form computer code for the code generation problem, and wherein the prefix code does not include or includes at least one code segment previously generated by the target machine learning model for the query; determining a first reward score corresponding to the first code segment based on an updated prefix code using a trained reward model, the first reward score indicating the correctness of the first code segment relative to the code generation problem, wherein the first code segment is appended to the prefix code to obtain the updated prefix code; and fine-tuning the target machine learning model by reinforcement learning, at least based on the first reward score.

[0004] In a second aspect of this disclosure, an apparatus for code generation is provided. The apparatus includes: a query receiving module configured to receive a query indicating a code generation problem; a first code segment generation module configured to generate a first code segment based on the query and prefix code using a target machine learning model, wherein the target machine learning model is built based on a language model and configured to iteratively generate code segments to form computer code for the code generation problem, and wherein the prefix code does not include or includes at least one code segment previously generated by the target machine learning model for the query; a first reward score determination module configured to determine a first reward score corresponding to the first code segment based on an updated prefix code using a trained reward model, the first reward score indicating the correctness of the first code segment relative to the code generation problem, wherein the first code segment is appended to the prefix code to obtain the updated prefix code; and a model fine-tuning module configured to fine-tune the target machine learning model through reinforcement learning, at least based on the first reward score.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions executable by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The medium stores a computer program that, when executed by a processor, performs the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes a computer program that, when executed by a processor, performs the method of the first aspect.

[0008] It should be understood that the description in the Summary of the Invention section of this invention is neither intended to identify key or essential features of the embodiments of this disclosure, nor to limit the scope of this disclosure. Other features of this disclosure will be readily envisioned from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure may be implemented is shown; Figure 2 A schematic diagram illustrating fine-tuning of target machine learning using a reward model according to some embodiments of the present disclosure is shown; Figure 3 A schematic diagram of a training reward model according to some embodiments of the present disclosure is shown; Figure 4 A schematic diagram illustrating a binary search of a code segment according to some embodiments of the present disclosure is shown; Figure 5 An algorithm for binary search for marking partial code prefixes according to some embodiments of this disclosure is shown; Figure 6 A flowchart of a process for code generation according to some embodiments of the present disclosure is shown; Figure 7 A block diagram of an apparatus for code generation according to some embodiments of the present disclosure is shown; and Figure 8 A block diagram of an electronic device in which one or more embodiments of the present invention may be implemented is shown. Detailed Implementation

[0010] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. Although some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0011] In the description of embodiments of this disclosure, the term "comprising" and similar terms will be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" will be understood as "at least partially based on". The term "one embodiment" or "the embodiment" will be understood as "at least one embodiment". The term "some embodiments" will be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the degree of matching between various data. For example, the aforementioned degree of matching can be obtained based on various currently available and / or future-developed technical solutions.

[0012] It should be understood that the data involved in this technical solution (including but not limited to the data itself, data acquisition or use) shall comply with the requirements of relevant laws, regulations and related provisions.

[0013] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be notified of the types, scope of use, and usage scenarios of the personal information involved in the present invention in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0014] For example, in response to receiving a user's active request, a prompt message is sent to the user, clearly informing them that the requested operation requires the acquisition and use of their personal information. Therefore, the user can choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media that perform the operations of the technical solutions disclosed herein.

[0015] As an optional but non-limiting implementation, a method for sending a prompt message to the user in response to a user's active request could be, for example, a pop-up window presenting the prompt message in text form. Furthermore, the pop-up window may also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0016] It is understood that the above notification and user authorization process is merely illustrative and does not limit the implementation of this disclosure. Other methods that comply with applicable laws and regulations may also be applied to the implementation of this disclosure.

[0017] As used herein, the term "model" learns the correlation between a given input and the output from training data, enabling the generation of a corresponding output for a given input after training is complete. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a model based on deep learning. As used herein, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably throughout this document.

[0018] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of input and output layers, and one or more hidden layers between them. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially such that the output of the previous layer is provided as the input to the next layer, where the input layer receives the input to the neural network, and the output layer's output is used as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the previous layer.

[0019] Machine learning typically comprises three phases: training, testing, and application (also known as inference). During training, a given model is trained using large-scale training data, iteratively updating parameter values ​​until the model can consistently infer from the training data that meets the expected objectives. Through training, the model learns the correlation between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​for the trained model are then determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process real-world inputs and determine the appropriate output based on the parameter values ​​obtained from training.

[0020] Figure 1 A block diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented is shown. Figure 1 In environment 100, code generation system 110 applies machine learning model 105 to perform code generation. Machine learning model 105 is configured to process query 112 input by user to generate code snippets 114.

[0021] Environment 100 also includes a reward model 116, which is configured to determine a reward score 118 corresponding to a code segment 114 generated by machine learning model 105. Machine learning model 105 is fine-tuned based at least on the reward score 118.

[0022] exist Figure 1 In this context, the code generation system 110 can be implemented on any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. Terminal devices can include any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, netbooks, tablet computers, media computers, multimedia tablet computers, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Servers include, but are not limited to, mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0023] It should be understood that Figure 1 The components and arrangements in the illustrated environment 100 are merely examples, and a computing system suitable for implementing the exemplary embodiments described in this disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown separately, the model training system 110 and the model application system 130 may be integrated into the same system or device. Implementations of this disclosure are not limited thereto.

[0024] It should be understood that the structure and function of each element in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0025] As mentioned above, unit tests can be used to generate code. However, the challenge stems from the nature of the reward signals derived from unit tests. These signals are inherently sparse, as they are only received at the end of a segment after the entire code has been generated and evaluated. This delay in feedback hinders learning efficiency and limits the model's ability to make incremental improvements during code generation. When an LM fails to generate code that passes any unit tests, it receives no meaningful learning signals, making it difficult to learn to solve more complex coding problems. Conversely, when a programmer's program fails unit tests, the programmer typically doesn't rewrite the code from scratch. Instead, they analyze the code to pinpoint and fix errors, using their understanding of the programming logic and structure to iteratively improve the current version. This incremental refinement process, involving receiving and acting on fine-grained feedback, is missing in current RL training loops that rely on unit test feedback for code generation.

[0026] The code generation task can be formulated within a reinforcement learning (RL) framework, where code generation is viewed as a series of decision-making steps. Once the language model has undergone supervised fine-tuning (SFT), an RL phase is employed to refine the model's ability to generate functionally correct code using feedback from unit tests. Unit test feedback is derived by executing the generated program against predefined test cases. This feedback serves as a signal for learning and can be converted into a reward. A simple reward function based on the unit test results can be defined as follows:

[0027] This binary reward formula encourages the language model to generate programs that can successfully pass all unit test cases. It generates a set of prompt words given unlabeled code. ,Model It is optimized to maximize the expected reward along all possible code generation trajectories.

[0028] While reinforcement learning derived from unit test feedback (RLTF) provides a framework for improving code generation models, it suffers significant limitations due to the sparsity of its reward signals. The binary nature of unit test feedback, which only indicates whether the entire program passes or fails, provides no guidance on which specific parts of the code contribute to the results. This lack of intermediate feedback makes it challenging for the model to identify and correct errors during training, leading to slow convergence and suboptimal performance. In contrast, human programmers iteratively develop and refine their code. When a program fails a unit test, they typically don't rewrite it from scratch. Instead, they analyze the code to pinpoint and fix errors, leveraging their understanding of the programming logic and structure. This incremental refinement process is crucial for effective problem-solving.

[0029] RL with unit test feedback enhances code generation for language models, but its reliance on sparse rewards provided only after code evaluation limits learning efficiency and incremental improvement. When generated code fails all unit tests, learning signals are not received, hindering progress on complex tasks.

[0030] To address at least some of the aforementioned problems, embodiments of this disclosure propose an improved solution for code generation. In this solution, a query indicative of a code generation problem is received. A target machine learning model is used to generate a first code segment based on the query and prefix code. The target machine learning model is built on a language model and configured to iteratively generate code segments to form computer code for the code generation problem. The prefix code does not include, or includes, at least one code segment previously generated by the target machine learning model for the query. A trained reward model is used to determine a first reward score corresponding to the first code segment based on the updated prefix code. The first reward score indicates the correctness of the first code segment relative to the code generation problem. The first code segment is appended to the prefix code to obtain the updated prefix code. The target machine learning model is fine-tuned through reinforcement learning, at least based on the first reward score.

[0031] Using these embodiments of the present disclosure, a reward model is integrated into an RL training framework for code generation, and this reward model provides signals by offering segment-level feedback indicating the correctness of each code segment generated by the target machine learning model. In this way, the learning efficiency of the target machine learning model can be improved by providing immediate guidance during code generation.

[0032] Reference Figures 2 to 8 Example embodiments of this disclosure are described.

[0033] Now for reference Figure 2 This illustrates a schematic diagram 200 of fine-tuning target machine learning through a reward model according to some embodiments of the present disclosure. For example... Figure 2 As shown, query 205, indicating a code generation problem, has been received. In a code generation task, the code generation problem can be defined as a lexical sequence... Each of them Indicates the first input prompt word Each element or term can include a question description. In the example, query 205 could be "Please sort a list of numbers using Python".

[0034] After query 205 is received, a target machine learning model (not shown) is used to generate a first code segment 210 based on query 205 and a prefix code. The target machine learning model is built on a language model and is configured to iteratively generate code segments to form computer code for a code generation problem. The prefix code does not include, or includes, at least one code segment previously generated by the target machine learning model for the query. In some embodiments, a code segment may refer to each line of code, two or more lines of code. In some embodiments, a code segment may refer to one or more code lexical units. The code segment is a unit for reward calculation or a unit for feedback signal acquisition.

[0035] In some embodiments, the primary objective of the target machine learning model is to process a given input query 205 (represented as...). This generates a consistent and syntactically correct sequence of code lexical units (also known as the first code segment 210). This sequence can be represented as... ,in This represents the total number of code generation steps, with each step corresponding to a code segment. Each individual code generation step... , Composed of a series of lexical elements Composition, in which Corresponding to the The first step within the [number] steps Each word element, and This indicates the number of lexical units in this step.

[0036] In some embodiments, for a given code generation problem (also known as query 205) In the case of, it is represented as The target machine learning model was used for the code generation step (also known as the first code segment 210). We model the conditional probability distribution, which is mathematically represented as... ,Depend on Parameterization. This can be achieved by using a dataset containing prompt word pairs (e.g., queries indicating code generation problems) and their corresponding code solutions. The training process optimizes the target machine learning model. This training process, often referred to as supervised fine-tuning (SFT), involves maximizing the log-likelihood of the dataset.

[0037] After the first code segment 210 is generated, a trained reward model 215 is used to determine a first reward score 220 corresponding to the first code segment 210 based on the updated prefix code. The first reward score 220 indicates the correctness of the first code segment 210 with respect to the code generation problem. The first code segment 210 is appended to the prefix code to obtain the updated prefix code. In some embodiments, a process reward model (PRM, as an example of reward model 215) may be introduced to evaluate the correctness of each line of code (i.e., each code segment) during the generation process. The PRM may act as an oracle that provides intermediate rewards based on the probability that the current code prefix is ​​expanded into a correct procedure.

[0038] In the following text, reference will be made to Figure 3 Describe how to train the reward model 215. Figure 3 A schematic diagram 300 of a training reward model 215 according to some embodiments of the present disclosure is shown.

[0039] In some embodiments, training data samples may be obtained for the reward model 215 to be trained. The training data samples may include sample queries 305 and sample computer code 310 for the sample queries 305. The sample computer code 310 may include multiple sample code segments having corresponding labeled reward scores 315. For example, the training data samples may be represented as... )in Indicates a sample query. Indicates until the Multiple sample code segments of a sample code segment, and This represents the corresponding labeled reward score for multiple sample code segments.

[0040] To effectively train the reward model 215, it may be necessary to provide a dataset with fine-grained annotations indicating the correctness of partial code sequences (also known as code segments). Correctness can be represented by labeled reward scores. Manually annotating the correctness of each line of code generated by the target machine learning model is expensive and not scalable. Instead, the ability of the target machine learning model is utilized to generate completions for partial code prefixes, and automated tests are used to evaluate their correctness. The key idea is to determine whether a partial code prefix (e.g., a sample code segment) can be extended to a complete program that passes all unit tests. If so, the prefix can be considered potentially correct; otherwise, it can be marked as incorrect. Thus, a corresponding labeled reward score 315 can be obtained.

[0041] In the following text, you can refer to Figure 4 To describe how to obtain training data samples, Figure 4A schematic diagram 400 illustrates a line-level binary search of code segments according to some embodiments of the present disclosure. In some embodiments, sample prefix code can be extracted from the sample computer code for a first reference code segment in the sample computer code. The sample prefix code may include the first reference code segment and at least one sample code segment preceding the first reference code segment in the sample computer code. In one example, such as Figure 4 As shown, there are 5 code segments in the sample computer code. The midpoint can be selected, namely code segment 430 (represented as...). This is designated as the first reference code segment. It includes the first reference code segment (i.e., code segment 430) and at least one sample code segment in the sample computer code preceding the first reference code segment (i.e., denoted as...). Code segment 410 and represented as The sample prefix code of code segment 420 can be extracted.

[0042] After the sample prefix code is extracted, code completion can be performed on the sample prefix code to obtain at least one predictive computer code. Then, a first labeled reward score corresponding to the first reference code segment can be determined based on the corresponding test results of a second set of multiple unit test cases applied to the at least one predictive computer code. In some embodiments, a cue word is given. (e.g., sample prefix code), using the current strategy of the target machine learning model. Generate a complete code response (as an example of at least one predicted computer code). In some embodiments, a code prefix (also known as a first reference code segment) may be determined for each section. against The correctness of the sample code. To achieve this, a K-optimal sampling strategy can be used to approximate an oracle capable of completing the sample prefix code including the first reference code segment. For each partial code prefix... It can use the current strategy of the target machine learning model to generate A potential complement It can be used for unit tests. Forming and executing a complete procedure These unit tests are applied to at least one predictive computer code.

[0043] For example, in some embodiments, a first marked reward score can be determined to indicate that the prefix code including the first reference code segment in the sample computer code is correct, based on the corresponding test results of a second set of multiple unit test cases that indicate success applied to one of the at least one predictive computer code. If any part of the complete program (i.e., the program including at least one predictive computer code) passes all unit tests, the first reference code segment can be marked as correct (e.g., marked with +1) to indicate that the prefix code including the first reference code segment in the sample computer code is correct.

[0044] In some embodiments, a first flagged reward score may be determined to indicate a failed test result based on the corresponding test results of a second set of multiple unit test cases applied to at least one predictive computer code, to indicate that the prefix code including a first reference code segment in the sample computer code is incorrect. If all of these programs fail all unit tests, the first reference code segment may be flagged as incorrect (e.g., flagged as -1) to indicate that the prefix code including the first reference code segment in the sample computer code is incorrect.

[0045] In some embodiments, based on determining that the prefix code of the first reference code segment in the sample computer code is correct according to a first marked reward score, at least one marked reward score corresponding to at least one sample code segment preceding the first reference code segment can be determined to indicate that the corresponding prefix code is correct. If any of these programs passes all unit tests and the first reference code is marked as correct (e.g., marked as +1), then at least one sample code segment preceding the first reference code segment can also be marked as correct (e.g., marked as +1) to indicate that the corresponding prefix code is correct. In the example, such as Figure 4 As shown, when the first reference code segment is code segment 430, code segments 410, 420, and 430 (i.e., the sample code segments preceding the first reference code segment and the first reference code segment) are marked as correct until the prefix code of the first reference code segment can be completed to pass all unit tests. When the first reference code segment is code segment 440, code segment 440 is marked as incorrect until the prefix code of the first reference code segment cannot be completed to pass all unit tests. For each part of the code prefix... Labels are assigned based on the results of the completed attempts. :

[0046] in Indicates unit testing A complete procedure that is formed and executed.

[0047] In some embodiments, based on determining that the prefix code of the first reference code segment in the sample computer code is correct, a second reference code segment can be selected from the suffix code following the first reference code segment in the sample computer code.

[0048] In some embodiments, the first reference code segment may be selected as the midpoint code segment in the sample computer code. The second reference code segment may be selected as the midpoint code segment in the suffix code. In some examples, where there are 10 code segments in the sample computer code, the midpoint (i.e., the fifth code segment) is selected as the first reference code segment. If the first reference code segment is correctly identified, the second reference code segment can be selected from the suffix code (i.e., the sixth to tenth code segments) of the sample computer code following the first reference code segment. In the example, the midpoint of the suffix code (i.e., the eighth code segment) may be selected as the second reference code segment.

[0049] In some embodiments, based on the determination that the prefix code of the first reference code segment in the sample computer code, which includes a first marked reward score, is incorrect, a second reference code segment can be selected from the partial prefix codes of the sample computer code that precede the first reference code segment and do not have a marked reward score. In some examples, if the first reference code segment is marked as incorrect, the error may originate from the partial prefix code of the sample computer code preceding the first reference code segment. A second reference code segment that does not have a marked reward score can be selected from the partial prefix codes.

[0050] In some embodiments, the second reference code segment can be selected as the midpoint code segment in the partial prefix code. In some examples, where there are 10 code segments in the sample computer code, the midpoint (i.e., the fifth code segment) is selected as the first reference code segment. If the first reference code segment is marked as incorrect, the second reference code segment can be selected from the partial code preceding the first reference code segment of the sample computer code (i.e., the first to fourth code segments). In the example, the midpoint of the partial code segment (i.e., the third code segment) that does not have a marked reward score can be selected as the second reference code segment. To effectively identify the transition point where the error occurs, a binary search of the code generation steps (e.g., the sample computer code) can be employed, such as... Figure 5 As shown, Figure 5 An algorithm for binary search for marking partial code prefixes, according to some embodiments of this disclosure, is illustrated. For example... Figure 5As shown, the midpoint code segment in the generated code segment can first be selected. Then, code completion can be performed on the prefix code including the midpoint code segment and the code segment preceding the midpoint code segment to obtain a complete program including at least one piece of predictive computer code. If the complete program passes all unit tests, the midpoint code segment is marked as correct, and the algorithm operates on the range of the midpoint code segment and the upper bound of the generated code segment. Otherwise, the midpoint code segment is marked as incorrect, and the algorithm operates on the range of the midpoint code segment and the lower bound of the generated code segment.

[0051] In this way, by using a binary search method, the efficiency of labeling each code segment generated by the target machine learning model can be improved.

[0052] After the second reference code segment is selected, a second tagged reward score corresponding to the second reference code segment can be determined. In some embodiments, the second tagged reward score can be determined based on unit tests of the code including the second reference code segment.

[0053] During the training of reward model 215, after acquiring training data samples, multiple predicted reward scores corresponding to multiple sample code segments can be generated separately using the reward model 215 to be trained. Using the acquired training data samples, the reward model 215 to be trained can predict the correctness (also known as predicted reward scores) of multiple sample code segments in the training data samples. Reward model 215 learns to assign higher rewards to sample code segments labeled as correct and lower rewards to sample code segments labeled as incorrect.

[0054] After the predicted reward scores are generated, the reward model can be trained based on the differences between the corresponding labeled reward scores corresponding to multiple sample code segments and the multiple predicted reward scores to obtain a trained reward model.

[0055] In some embodiments, the loss value of a predefined loss function can be determined based on the corresponding differences between the labeled reward scores corresponding to multiple sample code segments and multiple predicted reward scores, and the reward model can be trained to minimize or reduce the loss value of the predefined loss function. The training objective of minimizing or reducing the loss value can be expressed as follows:

[0056] in Represents multiple predicted reward scores and This indicates the corresponding marked reward score. This represents the difference between multiple predicted reward scores and their corresponding labeled reward scores. In some examples, mean squared error (MSE) loss or cross-entropy loss can be used as the loss function.

[0057] After the initial reward score is determined, the target machine learning model is fine-tuned via reinforcement learning, based at least on that initial reward score. In some embodiments, by providing intermediate feedback (e.g., the initial reward score), the PRM helps the target machine learning model identify and enhance beneficial code generation patterns while suppressing those that introduce errors. This fine-grained feedback reflects human coding practices, in which programmers continuously evaluate and adjust their code. In this way, the reward model can be trained well and can provide more accurate reward scores corresponding to the generated code snippets.

[0058] In some embodiments, the model parameter set of the value model can be initialized using the model parameter set of the trained reward model 215. The value model can be used for reinforcement learning. Return to Reference Figure 2 The set of values, 225, is output by the value model. The reward model uses a fixed prefix. The method of using a strategy to sample correct responses to annotate code can be considered as... The value estimation of the trained reward model 215 can serve as a useful inductive bias for initializing the value function (also known as the value model) in RL algorithms, which can alleviate the credit allocation problem by providing a more informative starting point.

[0059] In some embodiments, the trained reward model 215 can be used to provide step-level reward signals that guide more efficient policy exploration during RL training. By rating the correctness of each line in the code response (as an example of a code segment generated by the target machine learning model), the trained reward model 215 provides dense rewards that encourage the target machine learning model's policy to explore more promising code paths, thereby improving the performance of the target machine learning model.

[0060] In some embodiments, computer code for the code generation problem can be obtained. The computer code includes multiple code segments generated iteratively by a target machine learning model in response to a query. The overall reward score for the computer code can be generated based on the corresponding test results of a first set of multiple unit test cases applied to the computer program. In one example, as shown in Equation (1), the overall reward score for the computer code is 1 if it passes all the unit test cases applied to the computer program, and 0 otherwise.

[0061] After the overall reward score is generated, the target machine learning model can be fine-tuned through reinforcement learning based on the corresponding reward scores generated by the reward model 215 for multiple code segments and the overall reward score.

[0062] In some embodiments, the base model of the target machine learning model is initially fine-tuned on a Supervised Fine-Tuning (SFT) dataset and then used as initialization for subsequent RLHF training phases. Approximate Policy Optimization (PPO) can be used on the RLHF dataset. The above is a finely tuned base model (represented as) Fine-tuning is performed to obtain the model (represented as) In this disclosure, two types of outcome reward models (ORMs) can be used as objective functions for training RL. For non-encoded cue words, a general reward model derived from preference learning from a human-annotated dataset can be used. For encoded cue words, the ORM (e.g., reward model 215) can be defined as whether the response passes all unit tests. The binary indicator. The RLHF optimization objective can be defined as:

[0063] in Used for non-encoded prompt words and Used for encoding prompt words.

[0064] To ensure that the training data for the PRM (e.g., reward model 215) effectively covers the state space that the language model (e.g., objective machine learning model) might encounter during the next RL training phase, the policy model can be sampled from various RL phases. Specifically, four checkpoints are selected at uniform intervals throughout the training process of the RL baseline model. For each checkpoint, a sample is generated for the training dataset. Each encoded prompt word pair Each response is sampled. For each sampled response, padding for each partial code prefix can be applied. Figure 5 The algorithm described in the text describes the binary search labeling process. Then, the data collected from all checkpoints is aggregated into the PRM training set (denoted as: In the aggregated dataset, the PRM (e.g., reward model 215) can be initialized using a value model from the RL baseline, and the objective function defined by equation (3) is applied. Make minor adjustments.

[0065] As described above, regarding the integration of reward model 215 into RL training, reward model 215 can be used as a dense reward signal source and is used to initialize the value model. In the method of providing dense reward signals, reward model 215 assigns an additional reward signal at each line-end word (\n) in the code response for the encoded prompt word. Therefore, the RL optimization objective for the encoded prompt word is modified as defined below. and Weighted sum:

[0066] in The relative importance of the control reward model 215 in shaping rewards is considered. In one example, this can be done if the code response fails all unit tests (i.e., Set when =0) And it can pass all unit tests in the response (i.e., Set when =1) The intuitive understanding behind this reward shaping is to use the reward model 215 to provide informative signals when the RL policy fails to generate an effective solution, while minimizing the risk of over-optimization of the reward model once the correct solution is found.

[0067] Some implementations propose a PRM (e.g., reward model 215) that delivers dense line-level feedback on code correctness during generation, simulating human code refinement and providing immediate guidance.

[0068] Several implementations provide a comprehensive analysis of how PRM can be integrated into RL training for code generation. Various strategies for training robust code PRM are explored, and different methods for leveraging PRM to improve code generation performance are investigated. Based on experiments, practical solutions for using PRM in the context of code generation problems and integrating PRM into RL training are provided. Furthermore, using PRM simultaneously as both a dense reward and a value function initialization during RL training significantly improves code generation performance.

[0069] Some embodiments propose an efficient method that automatically generates process-level supervision data by identifying the first erroneous line in the generated code using a binary search. A Process Management Registry (PRM) can then be trained on this data to generate dense signals during RL training. The PRM can allow RL to benefit from unit test feedback during code generation.

[0070] Several implementations underwent systematic experiments to determine how to correctly and effectively integrate PRM into RL. Various strategies for training high-quality code using PRM and leveraging PRM to improve code generation were explored. Therefore, practical solutions for successfully using PRM in the context of code generation are provided. By following this solution, the pass rate of the target machine learning model was enhanced. Additionally, it was found that integrating PRM into RL training is beneficial for code generation in long-term scenarios.

[0071] Figure 6 A flowchart of a process 600 for code generation according to some embodiments of the present disclosure is shown. Process 600 can be performed in... Figure 1 The code generation system is implemented in 110 places.

[0072] At box 610, code generation system 110 receives a query indicating a code generation problem.

[0073] At box 620, the code generation system 110 uses a target machine learning model to generate a first code segment based on a query and a prefix code, wherein the target machine learning model is built based on a language model and configured to iteratively generate code segments to form computer code for a code generation problem, and wherein the prefix code does not include code segments previously generated by the target machine learning model for the query or includes at least one code segment previously generated by the target machine learning model for the query.

[0074] At box 630, the code generation system 110 uses a trained reward model to determine a first reward score corresponding to the first code segment based on the updated prefix code. The first reward score indicates the correctness of the first code segment relative to the code generation problem, wherein the first code segment is appended to the prefix code to obtain the updated prefix code.

[0075] At box 640, the code generation system 110 fine-tunes the target machine learning model through reinforcement learning, based at least on the first reward score.

[0076] In some embodiments, fine-tuning the target machine learning model by reinforcement learning based at least on a first reward score includes: initializing a model parameter set of a value model using a model parameter set of a trained reward model, wherein the value model is used for reinforcement learning; and fine-tuning the target machine learning model together with the value model by reinforcement learning based at least on the first reward score.

[0077] In some embodiments, process 600 further includes: obtaining computer code for a code generation problem, the computer code including a plurality of code segments iteratively generated by a target machine learning model for a query; generating an overall reward score for the computer code based on the corresponding test results of a first set of a plurality of unit test cases applied to a computer program; and wherein fine-tuning the target machine learning model by reinforcement learning based at least on a first reward score includes: fine-tuning the target machine learning model by reinforcement learning based on the corresponding reward scores generated by the reward model for the plurality of code segments and the overall reward score.

[0078] In some embodiments, process 600 further includes: acquiring training data samples for a reward model to be trained, the training data samples including sample queries and sample computer code for the sample queries, the sample computer code including a plurality of sample code segments having corresponding labeled reward scores; using the reward model to be trained, generating a plurality of predicted reward scores corresponding to the plurality of sample code segments respectively; and training the reward model based on the corresponding differences between the corresponding labeled reward scores corresponding to the plurality of sample code segments and the plurality of predicted reward scores to obtain a trained reward model.

[0079] In some embodiments, obtaining training data samples includes: extracting sample prefix code from the sample computer code for a first reference code segment in the sample computer code, the sample prefix code including the first reference code segment and at least one sample code segment in the sample computer code preceding the first reference code segment; performing code completion on the sample prefix code to obtain at least one predictive computer code; and determining a first labeled reward score corresponding to the first reference code segment based on the corresponding test results of a second set of multiple unit test cases applied on the at least one predictive computer code.

[0080] In some embodiments, determining a first tagged reward score includes: determining a first tagged reward score to indicate that the prefix code including the first reference code segment in the sample computer code is correct, based on test results indicating success for a second set of multiple unit test cases applied to at least one predictive computer code; and determining a first tagged reward score to indicate that the prefix code including the first reference code segment in the sample computer code is incorrect, based on test results indicating failure for a second set of multiple unit test cases applied to at least one predictive computer code.

[0081] In some embodiments, obtaining training data samples further includes: determining at least one labeled reward score corresponding to at least one sample code segment preceding the first reference code segment to indicate that the prefix code of the first reference code segment in the sample computer code is correct, based on determining that the first labeled reward score indicates that the prefix code of the first reference code segment is correct.

[0082] In some embodiments, obtaining training data samples further includes: selecting a second reference code segment from suffix codes following the first reference code segment in the sample computer code, based on determining that the prefix code of the first reference code segment in the sample computer code, which indicates that the first labeled reward score is correct; selecting a second reference code segment from a portion of the prefix code of the sample computer code, which precedes the first reference code segment and does not have a labeled reward score, based on determining that the prefix code of the first reference code segment in the sample computer code, which indicates that the first labeled reward score is incorrect; and determining a second labeled reward score corresponding to the second reference code segment.

[0083] In some embodiments, the first reference code segment is selected as the midpoint code segment in the sample computer code, and / or the second reference code segment is selected as the midpoint code segment in the suffix code or part of the prefix code.

[0084] Figure 7 A block diagram of an apparatus 700 for code generation according to some embodiments of the present disclosure is shown. The apparatus 700 may, for example, be in... Figure 1 The code generation system 110 is implemented or included in the code generation system 110. Various modules / components in the device 700 can be implemented by hardware, software, firmware, or any combination thereof.

[0085] As shown in the figure, the apparatus 700 includes: a query receiving module 710 configured to receive a query indicating a code generation problem; a first code segment generation module 720 configured to generate a first code segment based on the query and prefix code using a target machine learning model, wherein the target machine learning model is built based on a language model and configured to iteratively generate code segments to form computer code for the code generation problem, and wherein the prefix code does not include or includes at least one code segment previously generated by the target machine learning model for the query; a first reward score determination module 730 configured to determine a first reward score corresponding to the first code segment based on the updated prefix code using a trained reward model, the first reward score indicating the correctness of the first code segment relative to the code generation problem, wherein the first code segment is appended to the prefix code to obtain the updated prefix code; and a model fine-tuning module 740 configured to fine-tune the target machine learning model through reinforcement learning at least based on the first reward score.

[0086] In some embodiments, the model fine-tuning module 740 is further configured to initialize the model parameter set of the value model using the model parameter set of the trained reward model, wherein the value model is used for reinforcement learning; and to fine-tune the target machine learning together with the value model through reinforcement learning, at least based on a first reward score.

[0087] In some embodiments, the apparatus 700 further includes an overall reward score, configured to acquire computer code for a code generation problem, the computer code including multiple code segments iteratively generated by a target machine learning model for a query; generate an overall reward score for the computer code based on the corresponding test results of a first set of multiple unit test cases applied to a computer program; and wherein fine-tuning the target machine learning model by reinforcement learning based at least on a first reward score includes: fine-tuning the target machine learning model by reinforcement learning based on the corresponding reward scores generated by the reward model for the multiple code segments and the overall reward score.

[0088] In some embodiments, the apparatus 700 further includes a reward model training module configured to acquire training data samples for a reward model to be trained, the training data samples including sample queries and sample computer code for the sample queries, the sample computer code including a plurality of sample code segments having corresponding labeled reward scores; using the reward model to be trained, generating a plurality of predicted reward scores corresponding to the plurality of sample code segments respectively; and training the reward model based on the corresponding differences between the corresponding labeled reward scores corresponding to the plurality of sample code segments and the plurality of predicted reward scores to obtain a trained reward model.

[0089] In some embodiments, the reward model training module is further configured to: extract sample prefix code from the sample computer code for a first reference code segment in the sample computer code, the sample prefix code including the first reference code segment and at least one sample code segment in the sample computer code preceding the first reference code segment; perform code completion on the sample prefix code to obtain at least one predictive computer code; and determine a first labeled reward score corresponding to the first reference code segment based on the corresponding test results of a second set of multiple unit test cases applied on the at least one predictive computer code.

[0090] In some embodiments, the reward model training module is further configured to determine a first tagged reward score to indicate that the prefix code including the first reference code segment in the sample computer code is correct, based on the test results indicating successful test results of a second set of multiple unit test cases applied to at least one predictive computer code; and to determine a first tagged reward score to indicate that the prefix code including the first reference code segment in the sample computer code is incorrect, based on the test results indicating failed test results of a second set of multiple unit test cases applied to at least one predictive computer code.

[0091] In some embodiments, the reward model training module is further configured to determine at least one labeled reward score corresponding to at least one sample code segment preceding the first reference code segment to indicate that the prefix code of the first reference code segment in the sample computer code is correct, based on determining that the first labeled reward score indicates that the prefix code of the first reference code segment in the sample computer code is correct.

[0092] In some embodiments, the reward model training module is further configured to: select a second reference code segment from suffix codes following the first reference code segment in the sample computer code based on determining that the prefix code of the first reference code segment in the sample computer code, which indicates that the first labeled reward score is correct; select a second reference code segment from partial prefix codes of the sample computer code, which precede the first reference code segment and does not have a labeled reward score, based on determining that the prefix code of the first reference code segment in the sample computer code, which indicates that the first labeled reward score is incorrect; and determine a second labeled reward score corresponding to the second reference code segment.

[0093] In some embodiments, the first reference code segment is selected as the midpoint code segment in the sample computer code, and / or the second reference code segment is selected as the midpoint code segment in the suffix code or part of the prefix code.

[0094] Figure 8 A block diagram of an electronic device 800 in which one or more embodiments of the present invention may be implemented is shown. It should be understood that... Figure 8 The electronic device 800 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 800 can be used, for example, to implement... Figure 1 The code generation system 110. Electronic device 800 can also be used to implement... Figure 7 Device 700.

[0095] like Figure 8As shown, electronic device 800 is in the form of a general-purpose computing device. Components of electronic device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. Processing unit 810 may be a physical or virtual processor and can execute various processes according to programs stored in memory 820. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 800.

[0096] Electronic device 800 typically includes various computer storage media. Such media can be any available media accessible to electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 820 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or any combination thereof. Storage device 830 can be any removable or non-removable media and may include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (such as training data for training) and can be accessed within electronic device 800.

[0097] Electronic device 800 may also include additional removable / non-removable, volatile / non-volatile storage media. Although Figure 8 Not shown, but disk drives for reading from or writing to removable non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 820 may include computer program product 825 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0098] The communication unit 840 communicates with other computing devices via a communication medium. Furthermore, the functionality of the components in the electronic device 800 can be implemented by a single computing cluster or multiple computing machines, which can communicate via communication connections. Therefore, the electronic device 800 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0099] Input device 850 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 860 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 800 can also communicate with one or more external devices (not shown) as needed via communication unit 840. External devices (such as storage devices, display devices, etc.) communicate with one or more devices that enable the user to interact with electronic device 800, or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interfaces (not shown).

[0100] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions or a computer program thereon, which are executed by a processor to implement the above-described method. According to an exemplary implementation of this disclosure, a computer program product is also provided. This computer program product is physically stored on a non-transitory computer-readable medium and includes computer-executable instructions that are executed by a processor to implement the above-described method.

[0101] This document describes various aspects of the disclosure with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of each block in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0102] These computer-readable program instructions may be provided to the processing unit of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine that generates devices to implement the functions / actions specified in one or more boxes of a flowchart and / or block diagram when executed by the processing unit of the computer or other programmable data processing apparatus. These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions enable a computer, programmable data processing apparatus, and / or other apparatus to function in a particular manner. Therefore, a computer-readable medium containing the instructions includes a product comprising instructions for implementing aspects of the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0103] Computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus or other device to enable the execution of a series of operational steps on the computer, other programmable data processing apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable data processing apparatus or other device, perform the functions / actions specified in one or more boxes in a flowchart and / or block diagram.

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate possible architectures, functions, and operations of systems, methods, and computer program products implemented according to this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions marked in the blocks may occur in an order different from that marked in the drawings. For example, two consecutive blocks may actually be executed in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system performing the specified function or action, or by a combination of dedicated hardware and computer instructions.

[0105] Each embodiment of this disclosure has been described above. The above description is exemplary and not exhaustive, and is not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements of each embodiment in the market, or to enable others skilled in the art to understand the various embodiments disclosed herein.

Claims

1. A method for code generation, comprising: Receive queries regarding instruction code generation issues; Using a target machine learning model, a first code segment is generated based on the query and a prefix code, wherein the target machine learning model is built based on a language model and configured to iteratively generate code segments to form computer code for the code generation problem, and wherein the prefix code does not include code segments previously generated by the target machine learning model for the query or includes at least one code segment previously generated by the target machine learning model for the query; Using a trained reward model, a first reward score corresponding to the first code segment is determined based on the updated prefix code. The first reward score indicates the correctness of the first code segment relative to the code generation problem, wherein the first code segment is appended to the prefix code to obtain the updated prefix code. as well as The target machine learning model is fine-tuned through reinforcement learning, based at least on the first reward score.

2. The method of claim 1, wherein fine-tuning the target machine learning model through reinforcement learning based at least on the first reward score comprises: The model parameter set of the value model is initialized using the model parameter set of the trained reward model, wherein the value model is used for the reinforcement learning; as well as Based at least on the first reward score, the target machine learning is fine-tuned together with the value model through the reinforcement learning.

3. The method according to claim 1 or 2, further comprising: Obtain the computer code for the code generation problem, the computer code comprising multiple code segments iteratively generated by the target machine learning model in response to the query; Based on the corresponding test results of the first set of multiple unit test cases applied to the computer program, an overall reward score is generated for the computer code. as well as The fine-tuning of the target machine learning model through reinforcement learning based on at least the first reward score includes: Based on the corresponding reward scores generated by the reward model for the multiple code segments and the overall reward score, the target machine learning model is fine-tuned through reinforcement learning.

4. The method according to any one of claims 1 to 3, further comprising: Obtain training data samples for the reward model to be trained, the training data samples including sample queries and sample computer code for the sample queries, the sample computer code including multiple sample code segments with corresponding labeled reward scores; Using the reward model to be trained, generate multiple predicted reward scores corresponding to the multiple sample code segments respectively; as well as The reward model is trained based on the corresponding differences between the labeled reward scores corresponding to the plurality of sample code segments and the plurality of predicted reward scores to obtain the trained reward model.

5. The method according to claim 4, wherein obtaining the training data samples comprises: Regarding the first reference code segment in the sample computer code, Extract sample prefix code from the sample computer code, the sample prefix code including the first reference code segment and at least one sample code segment in the sample computer code preceding the first reference code segment; Code completion is performed on the sample prefix code to obtain at least one prediction computer code; as well as Based on the corresponding test results of the second set of multiple unit test cases applied on the at least one predictive computer code, a first marked reward score corresponding to the first reference code segment is determined.

6. The method of claim 5, wherein determining the first marked reward score comprises: Based on the test results indicating successful test results of the corresponding test results of the second set of multiple unit test cases applied to one of the at least one predictive computer codes, the first marked reward score indicating that the prefix code including the first reference code segment in the sample computer code is correct is determined; as well as Based on the corresponding test results indicating failed test results from the second set of multiple unit test cases applied on the at least one predictive computer code, the first marked reward score indicating that the prefix code including the first reference code segment in the sample computer code is incorrect is determined.

7. The method according to claim 6, wherein obtaining the training data samples further comprises: Based on determining that the first marked reward score indicates that the prefix code of the first reference code segment in the sample computer code is correct, at least one marked reward score corresponding to the at least one sample code segment preceding the first reference code segment is determined to indicate that the corresponding prefix code is correct.

8. The method according to claim 6, wherein obtaining the training data samples further comprises: Based on the determination that the prefix code of the first reference code segment in the sample computer code is correct, a second reference code segment is selected from the suffix code following the first reference code segment in the sample computer code; Based on the determination that the prefix code of the first reference code segment in the sample computer code is incorrect, a second reference code segment is selected from the partial prefix code of the sample computer code, the partial prefix code being prior to the first reference code segment and not having a marked reward score; as well as Determine the second marked reward score corresponding to the second reference code segment.

9. The method according to claim 5 or 8, wherein the first reference code segment is selected as the midpoint code segment in the sample computer code, and / or The second reference code segment is selected as the midpoint code segment in the suffix code or the partial prefix code.

10. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit, stores instructions executable by the at least one processing unit, which, when executed by the at least one processing unit, cause the device to perform actions, including: Receive queries regarding instruction code generation issues; Using a target machine learning model, a first code segment is generated based on the query and a prefix code, wherein the target machine learning model is built based on a language model and configured to iteratively generate code segments to form computer code for the code generation problem, and wherein the prefix code does not include code segments previously generated by the target machine learning model for the query or includes at least one code segment previously generated by the target machine learning model for the query; Using a trained reward model, a first reward score corresponding to the first code segment is determined based on the updated prefix code. This first reward score indicates the correctness of the first code segment relative to the code generation problem, wherein the first code segment is appended to the prefix code to obtain the updated prefix code; and The target machine learning model is fine-tuned through reinforcement learning, based at least on the first reward score.

11. The electronic device of claim 10, wherein fine-tuning the target machine learning model via reinforcement learning based at least on the first reward score comprises: The model parameter set of the value model is initialized using the model parameter set of the trained reward model, wherein the value model is used for the reinforcement learning; as well as Based at least on the first reward score, the target machine learning is fine-tuned together with the value model through the reinforcement learning.

12. The electronic device according to claim 10 or 11, wherein the action further comprises: Obtain the computer code for the code generation problem, the computer code comprising multiple code segments iteratively generated by the target machine learning model in response to the query; Based on the corresponding test results of the first set of multiple unit test cases applied to the computer program, an overall reward score is generated for the computer code. as well as The target machine learning model is fine-tuned through reinforcement learning based on at least the first reward score, including: Based on the corresponding reward scores generated by the reward model for the multiple code segments and the overall reward score, the target machine learning model is fine-tuned through reinforcement learning.

13. The electronic device according to any one of claims 10 to 12, wherein the operation further comprises: Obtain training data samples for the reward model to be trained, the training data samples including sample queries and sample computer code for the sample queries, the sample computer code including multiple sample code segments with corresponding labeled reward scores; Using the reward model to be trained, generate multiple predicted reward scores corresponding to the multiple sample code segments respectively; as well as The reward model is trained based on the corresponding differences between the labeled reward scores corresponding to the plurality of sample code segments and the plurality of predicted reward scores to obtain the trained reward model.

14. The electronic device of claim 13, wherein acquiring the training data sample comprises: Regarding the first reference code segment in the sample computer code, Extract sample prefix code from the sample computer code, the sample prefix code including the first reference code segment and at least one sample code segment in the sample computer code preceding the first reference code segment; Code completion is performed on the sample prefix code to obtain at least one prediction computer code; as well as Based on the corresponding test results of the second set of multiple unit test cases applied on the at least one predictive computer code, a first marked reward score corresponding to the first reference code segment is determined.

15. The electronic device of claim 14, wherein determining the first marked reward score comprises: Based on the test results indicating successful test results of the corresponding test results of the second set of multiple unit test cases applied to one of the at least one predictive computer codes, the first marked reward score indicating that the prefix code including the first reference code segment in the sample computer code is correct is determined; as well as Based on the corresponding test results indicating failed test results from the second set of multiple unit test cases applied on the at least one predictive computer code, the first marked reward score indicating that the prefix code including the first reference code segment in the sample computer code is incorrect is determined.

16. The electronic device of claim 15, wherein acquiring the training data sample further comprises: Based on determining that the first marked reward score indicates that the prefix code of the first reference code segment in the sample computer code is correct, at least one marked reward score corresponding to the at least one sample code segment preceding the first reference code segment is determined to indicate that the corresponding prefix code is correct.

17. The electronic device of claim 15, wherein acquiring the training data samples further comprises: Based on the determination that the prefix code of the first reference code segment in the sample computer code is correct, a second reference code segment is selected from the suffix code of the sample computer code after the first reference code segment; Based on the determination that the prefix code of the first reference code segment in the sample computer code is incorrect, a second reference code segment is selected from the partial prefix code of the sample computer code, the partial prefix code being prior to the first reference code segment and not having a marked reward score; as well as Determine the second marked reward score corresponding to the second reference code segment.

18. The method of claim 14 or 17, wherein the first reference code segment is selected as a midpoint code segment in the sample computer code, and / or The second reference code segment is selected as the midpoint code segment in the suffix code or the partial prefix code.

19. A computer-readable storage medium having a computer program stored thereon, the computer program causing the electronic device, when executed by the electronic device, to perform the method according to any one of claims 1 to 9.

20. A computer program product implemented on a computer-readable medium and comprising computer-executable instructions that are executed by a processor to perform the method according to any one of claims 1 to 9.

Images