System and Method for Program Synthesis

Abstract

The embodiments described in this specification provide a reinforcement learning-based framework using a pre-trained language model (LM) for program synthesis tasks. Specifically, the framework adopts a training strategy that optimizes the pre-trained LM for program synthesis tasks in the actor-critic approach.

Description

[Technical Field]

[0001] [Cross reference] This international application claims priority under the concurrently pending U.S. Nonprovisional Applications No. 17 / 896,942 and No. 17 / 896,946, filed on 26 August 2022 by the same applicant, each of which is a nonprovisional application under the U.S. Provisional Application No. 63 / 344,900, filed on 23 May 2022, and which claims priority under Section 119 of the U.S. Patent Act.

[0002] All of the above applications are expressly incorporated herein by reference in their entirety.

[0003] [Technical field] The embodiments generally relate to machine learning systems, and more specifically to systems and methods for program synthesis through pre-trained models and deep reinforcement learning. [Background technology]

[0004] Program synthesis, also commonly referred to as code generation, is the task of generating computer code programs that satisfy a problem specification, such as sorting a list, merging two data tables, and / or similar tasks. When program synthesis is treated as a sequence-to-sequence task, some pre-trained language models can be adapted to take an input sequence as a natural language problem specification and then generate a sequence of code as an output program. However, these existing language models may have limited code generation performance because they often train the program synthesis model only from natural language problem descriptions and ground truth programs, following standard supervised fine-tuning procedures. Such a paradigm largely ignores some important but potentially useful signals in the problem specification, such as unit tests, resulting in poor performance when solving complex, unfamiliar coding tasks.

[0005] Therefore, an efficient and accurate program synthesis model is needed.

Brief Description of Drawings

[0006] [Figure 1] A simplified block diagram showing an exemplary architecture that employs an actor-critic framework to optimize a pre-trained language model (LM) (and a fine-tuned LM) for program synthesis according to the embodiments described herein.

[0007] [Figure 2] A simplified block diagram showing an exemplary program synthesis task according to an embodiment described herein.

[0008] [Figure 3] A simplified block diagram showing an example of a reinforcement learning-based program synthesis framework for an exemplary program synthesis task according to an embodiment described herein.

[0009] [Figure 4] A simplified block diagram showing an exemplary training procedure of the critic network of FIG. 1 according to the embodiments described herein.

[0010] [Figure 5] A simplified block diagram showing a critic sampling (CS) framework for program synthesis that uses the learned LM (actor network) from FIG. 1 during inference according to the embodiments described herein.

[0011] [Figure 6] A simplified diagram showing a computing device for implementing the reinforcement learning-based program synthesis framework illustrated in FIGS. 1-5 according to some embodiments.

[0012] [Figure 7] Figures 1 to 5 are simplified block diagrams showing a networked system suitable for implementing the program synthesis framework described herein and other embodiments described herein.

[0013] [Figure 8] Figure 1 is an exemplary logic flowchart illustrating a reinforcement learning-based training method for program synthesis based on the actor-critic framework shown, according to some embodiments described herein.

[0014] [Figure 9] Figure 5 is an exemplary logic flowchart illustrating a program synthesis method based on the LM shown, according to some embodiments described herein. [Figure 9B] Figure 5 shows an exemplary logic flow diagram illustrating a program synthesis method based on the LM shown in some embodiments described herein (continued).

[0015] [Figure 10] These are exemplary data tables or charts showing exemplary performance comparisons between the program synthesis frameworks described in Figures 1 to 9 and various baseline models, according to some embodiments described herein. [Figure 11] These are exemplary data tables or charts showing exemplary performance comparisons between the program synthesis frameworks described in Figures 1 to 9 and various baseline models, according to some embodiments described herein. [Figure 12] These are exemplary data tables or charts showing exemplary performance comparisons between the program synthesis frameworks described in Figures 1 to 9 and various baseline models, according to some embodiments described herein. [Figure 13] These are exemplary data tables or charts showing exemplary performance comparisons between the program synthesis frameworks described in Figures 1 to 9 and various baseline models, according to some embodiments described herein. [Figure 14] These are exemplary data tables or charts showing exemplary performance comparisons between the program synthesis frameworks described in Figures 1 to 9 and various baseline models, according to some embodiments described herein. [Figure 15] These are exemplary data tables or charts showing exemplary performance comparisons between the program synthesis frameworks described in Figures 1 to 9 and various baseline models, according to some embodiments described herein. [Figure 16] These are exemplary data tables or charts showing exemplary performance comparisons between the program synthesis frameworks described in Figures 1 to 9 and various baseline models, according to some embodiments described herein. [Figure 17] These are exemplary data tables or charts showing exemplary performance comparisons between the program synthesis frameworks described in Figures 1 to 9 and various baseline models, according to some embodiments described herein. [Figure 18] These are exemplary data tables or charts showing exemplary performance comparisons between the program synthesis frameworks described in Figures 1 to 9 and various baseline models, according to some embodiments described herein. [Figure 19] These are exemplary data tables or charts showing exemplary performance comparisons between the program synthesis frameworks described in Figures 1 to 9 and various baseline models, according to some embodiments described herein.

[0016] [Figure 20A] Figures 1 to 9 show examples of programming problems and corresponding programs generated by the program synthesis framework and various baseline models described herein, according to some embodiments. [Figure 20B]Figures 1 to 9 show examples of programming problems and corresponding programs generated by the program synthesis framework and various baseline models described herein, according to some embodiments. [Figure 20C] Figures 1 to 9 show examples of programming problems and corresponding programs generated by the program synthesis framework and various baseline models described herein, according to some embodiments. [Modes for carrying out the invention]

[0017] In the diagram, elements with the same name have the same or similar functions.

[0018] As used herein, the term “Network” may include any hardware or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented on or with it.

[0019] As used herein, the term “module” may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0020] Existing language models that can be used for program synthesis are often trained using the traditional next-token prediction (NTP) objective, which maximizes the likelihood of the next ground truth token. Training a model solely on the next-token prediction objective in a "supervisor-forced" manner often leads to cumulative errors during test time, especially when tokens are generated by conditioning on previously sampled tokens rather than ground truth tokens. This problem is more serious in the field of program synthesis, where existing token matching scores such as BLEU may have failed to measure the functional accuracy of a complete program.

[0021] Furthermore, existing language models sometimes fail to leverage potentially meaningful signals from unit tests, which directly determine the model's performance through the functional correctness of the program. Current approaches ignore these crucial signals during model optimization and generation procedures.

[0022] In view of the problems in existing program synthesis models, the embodiments described herein provide a reinforcement learning-based framework using a pre-trained language model (LM) for program synthesis tasks. Specifically, a pre-trained LM (e.g., pre-trained on publicly available code data) can be fine-tuned for program synthesis tasks for pairs of natural language problem descriptions and corresponding solution programs. The fine-tuned LM may then function as an actor network, which synthetically samples sequences generated from this actor to form sampled programs in response to inputs of the same problem description containing both correct and incorrect programs. These program samples are passed to a critic model trained as an error predictor to predict test outcomes of input programs given unit tests, which determines a return that evaluates the functional accuracy of these program samples. The returns generated from the critic model are then used to compute a policy gradient to minimize the expected (negative) return. The actor network is then fine-tuned based on the policy gradient.

[0023] In this way, the pre-trained LM is fine-tuned for the program synthesis task using a reinforcement learning approach. For example, the pre-trained parameters of the LM may function as a stochastic policy for the actor network, and actions may be generated accordingly as predictions for each token for the output program. The pre-trained LM (actor network) receives a return measured by the functional accuracy of the generated program, and the goal of reinforcement learning is to minimize the expected return.

[0024] In one embodiment, during inference, a finely tuned LM via the RL framework may be used to generate one or more code programs in response to a natural language problem description. To improve the accuracy and precision of the resulting programs, programming refinement and / or program repair procedures may be optionally used to refine and / or repair the generated programs based on the functional accuracy of the programs generated during test time. Specifically, exemplary unit testing and critical models are employed to filter and select "pass" programs (those that pass unit tests) and "fail" programs (those that fail unit tests), respectively, from the LM-generated programs.

[0025] The "pass" program can then be used to improve program generation. That is, a subsequence from the "pass" program is used as a "seed" to initialize and condition the LM model, resampling new tokens to obtain a new output program, and, for example, generating subsequent tokens following the "seed" to form the output program. In this way, the regenerated program, already conditioned for the "pass" subsequence, can yield a high likelihood of passing the unit test.

[0026] "Failed" programs can be used to repair program generation. From among the "failed" programs, those with a relatively high likelihood of passing unit tests (compared to other "failed" programs) may be selected. These selected program candidates are linked with their respective error information (e.g., whether the "failed" program failed to compile, run, or produce correct test results, or whether a specific error occurred, such as a syntax error). A program repair module can receive the linked input and generate an output code program. Thus, the regenerated (repaired) program is generated based on information about possible previous errors and therefore has a higher likelihood of being "repaired" and passing unit tests.

[0027] Figure 1 is a simplified block diagram showing an exemplary architecture 100 that employs an actor-critic framework 150 to optimize a pre-trained LM 110 (and a fine-tuned LM 120) for program synthesis, according to embodiments described herein. In one embodiment, the architecture 100 includes a pre-training / fine-tuning unit 145 for one or more LMs 110 and an actor-critic framework 150. Specifically, the program synthesis task may be formulated as a reinforcement learning (RL) problem so that the actor-critic framework 150 can apply an actor-critic reinforcement learning (RL) approach to improve the performance of a pre-trained LM 120 that has been fine-tuned for program synthesis from the pre-training / fine-tuning stage 145.

[0028] In one embodiment, at stage 145, LM 110 may first be pre-trained on publicly available code data 102 (e.g., from GitHub). For example, LM 110 may include a Transformer model as the backbone of the program synthesis system described herein. An example of such a pre-trained LM 110 may be a multilingual code recognition language model pre-trained on a large source code corpus curated from GitHub, such as CodeT5, described in concurrently pending and co-owned U.S. Nonprovisional Application No. 17 / 450,968, filed on 27 August 2021, which is expressly incorporated herein by reference in its entirety.

[0029] In one embodiment, the public code data 102 may include a Python pre-training dataset such as the GitHub code dataset. The public code data 102 may have publicly available non-personally identifiable information compiled from GitHub, consisting of permissively licensed Python code (e.g., "mit", "apache-2", "bsd-3-clause", "bsd-2-126 clause", "cc0-1.0", "unlicense", "isc"). The resulting Python dataset (GCPY) has 10.5 billion tokens, which is 10 times larger than the CodeSearchNet (CSN) corpus used in the original CodeT5 pre-training.

[0030] In one embodiment, the LM 110 may be pre-trained with a pre-training task similar to those used in CodeT5, such as Masked Span Prediction (MSP). While the MSP task is useful for code comprehension, it is significantly different from the objectives of program synthesis. To mitigate this gap, a Next Token Prediction (NTP) pre-training task may be used when pre-training the LM 110. Specifically, pivot positions are uniformly sampled for each code sample, and then the content preceding the pivot is passed to the LM 110's encoder, and the remainder to the LM 110's decoder. To control the length of the input and output sequences, pivots may be limited to within 10% to 90% of the original sequence.

[0031] After pre-training, the pre-trained LM 110 can be fine-tuned for specific program synthesis tasks. Following the sequence-to-sequence approach, program synthesis training pairs of a natural language problem description 105 in the form of an input sequence D and a corresponding solution code program 106 can be used to fine-tune the pre-trained LM 110. In response to the input sequence D, the pre-trained LM 110 outputs a program that can solve the problem.

number

number

[0032] Therefore, the model parameters θ of the pre-trained LM 110 can be fine-tuned during training time to maximize the likelihood of the ground truth criterion program. Specifically, W=(w1,…,w T If we consider ) as the ground truth program, the objective is the cross-entropy loss of 10⁸, that is,

number

number

number

[0033] In one embodiment, the finely tuned LM 120 is evaluated against unit tests 112 corresponding to the problem description. Each test includes a pair of inputs and ground truth outputs. In some exemplary real-world program synthesis tasks, exemplary unit tests are often provided as part of the problem specification.

[0034] In one embodiment, the finely tuned LM 120 is then passed to the actor-critic framework 150, which acts as the actor network 130. Specifically, the learned parameters θ of the finely tuned LM model 120 can be thought of as a probabilistic policy that determines an action as a prediction for each token in the sampled program 133 in response to the input of the problem description 105. Following each action, the LM model 120 (or synonymously, the actor network 130) updates its hidden state representation, which is used by the policy to determine the next action in the next decoding step. The generated tokens of the sampled program 133 may be sent to the critic network 140. At the end of the generation episode (i.e., <endoftext>Once the token is found), the actor network 130 receives the return r measured by the critic network 140 based on the functional accuracy of the generated program 133.

[0035] Specifically, for each token W t s in each synthetic sample array W s =(W1 s ,...,W t s ) sampled by the actor network 130 at the decoding time step t, the critic network 140 can determine the return by checking its functional accuracy. On the one hand, the problem description 105 is associated with one or more unit tests 112 including exemplary test inputs and corresponding outputs for solving the problem description 105. On the other hand, the generated program 133 is also passed to the compiler together with the corresponding unit tests 112. The generated program 133 is then compiled, executed using the test inputs from the unit tests 112, and execution results are generated. From the output of the execution, depending on whether the synthetic sample sequence W s can be fully compiled and executed, and if the execution is successful, depending on whether the execution results match the test outputs of the unit tests 112, the return r can be determined, that is,

Number

[0036] The determined reward r can then be used to calculate the reinforcement learning training objective of minimizing the expected return 135, that is,

Number

number

[0037] In one embodiment, a “baseline” program may be employed in the RL training of actor network 130. Specifically, a greedy decoding strategy may be used as the baseline, and any generated samples 133 that perform better than this baseline are given a positive return estimate, and otherwise a negative return estimate. This relative normalization technique allows the model to explore incomplete programs as long as their returns are better than those of the baseline. In other words, given a problem description 105, a baseline program sample sequence Wb may be generated using the baseline model. The baseline return r(W) b ) is r(W s The expected gradient estimate can be determined in a similar manner to the above, and the expected gradient estimate can be calculated to reflect whether the sampled program sequence performs better than the baseline program sequence by including the respective rewards.

number

[0038] Figure 2 is a simplified block diagram illustrating an exemplary program synthesis task according to one embodiment described herein. As shown in Figure 2, an example of a program synthesis task includes a problem specification 105 in natural language that describes the problem “not a palindrome” and “print the maximum length of a substring.” The corresponding solution program 106 includes a code segment that solves the problem described in specification 105. A unit test 112 may include an exemplary input-output test pair corresponding to problem specification 105, for example, if input = “wuffuw”, output (e.g., maximum length of a non-palindrome substring) = 5.

[0039] A pre-trained language model (LM 120) can be adapted to receive an input sequence as a problem specification 105 in natural language and generate a sequence of code as an output program. Once the problem specification 105 is passed to a code generator (such as a pre-trained and fine-tuned LM 130), the expected output is a program that is checked for functional correctness against unit tests 112.

[0040] Figure 3 is a simplified block diagram showing an example of a reinforcement learning-based program synthesis framework 300 for an exemplary program synthesis task according to one embodiment described herein. The RL-based program synthesis framework 300 illustrates the dynamics within the actor-critic framework 145 of Figure 1 in an RL manner. Specifically, in the RL network 300, a finely tuned LM may function as an actor that, in response to the input of the problem specification 105, determines an action 216 as a prediction for each token in the output program sequence, according to the learned parameters, i.e., the policy, of the finely tuned LM model θ. The action 216 may then be sent to a critic network 140 which serves as a value function for updating the actor 130 by calculating a value 217 (e.g., the policy gradient) based on the current action 216. The compiler 204 may also serve as the environment for the actor 130 and the critic 140, receiving an action 216, and generating a reward 213 by, for example, compiling and executing a sampled program sequence consisting of prediction tokens (actions 216) from actor 130, and comparing the execution result with a unit test 112. The execution state 214 of the environment (compiler 204) may be shared with actor 130 and the critic 140 so that each can make their own predictions.

[0041] Figure 4 is a simplified block diagram showing an exemplary training procedure for the Critic Network 140 of Figure 1 according to embodiments described herein. The Critic Model 140 includes a Sequence-to-Sequence model 402, linear and softmax operators 404, a maximum pooling module 406, and a return estimation module 408.

[0042] In one embodiment, the Critic Model 140 is a sampled program W from the problem description D 105 and the actor network 130 in Figure 1. s =(W1 s ,...,W t s The critique model is parameterized as a neural network having a parameter Φ that accepts input as ). The critique model may receive the problem specification 105 and program 133 or 134 as input sequences and then train an error predictor that predicts one of four possible test outcomes {CompileError;RuntimeError;FailedTest;PassedTest} as relatedly described in the reward definition.

[0043] For example, the Critic Model 140 may include a Transformer model smaller than the Actor Model 130 as the base architecture, i.e., a Sequence-to-sequence model 402. The context hidden state of the program token obtained from the Critic Model decoder {h1,…,h T The data is passed to the linear layer 404, which is then maxed out along the sequence length dimension via the max pooling layer 206. h pool =Pooling(Linear (h1),…,Linear(h T ))(6) Next, the critic's prediction of the unit test outcome is

number

[0044] In this way, the training objective 409 of the parameter Φ of the critique model 130 can be calculated as the cross-entropy loss between the predicted single-unit test outcome from the maximum pooling layer 406 of the critique model 130 and the ground truth single-unit test outcome 413. L critic (Φ) = -logpΦ(u│W) s ,D) (8) Here, u is the sampled program sequence W. s The ground truth unit test outcome 413, given by the compiler after passing 133 to the unit test 112 corresponding to the problem, is shown. The calculated training objective L critic (Φ) is then used to update the critical model 140 (e.g., the maximum pooling layer 406, the linear and softmax operators 404, and the Sequence-to-sequence model 402) via backpropagation.

[0045] After training Critic Model 140, regarding the Ground Truth Unit Test output

number

number

number

number

number

number

[0046] In some embodiments, to improve and stabilize the training process, a baseline program 134 is considered, for example, by passing it to a unit test 112 to generate a baseline test result 414. In this way, relative returns are generated by comparing the sample test result 413 with the baseline test result 414. Specifically, the return estimation module 408 may then calculate the policy gradient based on the intermediate returns (with the baseline test result 414 generated by passing the baseline program sequence 134 to the unit test 112).

number

[0047] In one embodiment, imitation learning may be employed to warm-start a pre-trained LM model 110 in Lce for up to 10 epochs. Then, a sampled program sequence is taken from this actor network 130 and used to train a critic model 140 while keeping the parameters of the actor network 130 frozen. For example, if the actor network is a CodeT5 actor model, a CodeT5-small architecture may be used for the critic model 140, and if the actor model is a GPT variant, a GPT2-small critic architecture may be used for the critic model 140.

[0048] In one embodiment, in addition to the synthetic program 133, the ground truth program 106 of the training samples may also be used to train the Critic network 140. These samples are considered complete programs and always have the label PassedTest. After training the Critic, both Lce and Lrl are applied with equal weights to fine-tune the actor network 130. To optimize the LM actor network 130, in each training optimization step, the expected gradient is set to a single sample W s ~p θ It can be approximated by this.

number

[0049] Figure 5 is a simplified block diagram showing a critical sampling (CS) framework 500 for program synthesis using a trained LM (Actor Network 130) from Figure 1 during inference, according to embodiments described herein. The LM 130, fine-tuned for the program synthesis task from Figure 1, can be used to generate, improve, and repair a program based on the results of exemplary unit tests of the corresponding problem. Specifically, a dual strategy (referred to as “critical sampling” (CS)) including a program repair procedure 560 and a programming improvement procedure 550 can be implemented to generate and improve a program during inference from both successful cases (program improvement) and unsuccessful cases (program repair). The program improvement procedure 550 and / or the program repair procedure 560 may be performed either separately or together (as indicated by the dotted lines in Figure 5) after the LM 130 has generated the output program 533.

[0050] In one embodiment, the test problem description 505 may be received in a finely tuned LM 130 during the inference phase. The exemplary unit test inputs and outputs provided in the input problem description 505 may be used to improve the generation procedure during inference. For example, exemplary input-output pairs may be extracted from the problem description 505 to form an exemplary unit test 112.

[0051] For each problem description 505, the finely tuned LM 130 can generate N programs 533. Each of the generated programs 533 can then be passed to an exemplary unit test, which is often embedded as part of the problem specification 505. Specifically, the generated programs 533 can be filtered by the results of the exemplary unit tests in the filtering module 535, and the filtering module 535 sets the programs that passed the exemplary tests.

number

number

[0052] The generated program 533 can undergo program improvement procedure 550 to generate the final program 555. Specifically, the pass set

number

[0053] In one implementation configuration, the set is acceptable.

number

number

number

[0054] Therefore, subsequence 543 is, <endoftext>The (actor) LM 130 is used as a seed 545 to initialize and condition it for resampling new tokens up to the token. In this iteration, each seed sequence can be stacked N / |P| times for upsampling. This results in the same number of output programs N as the initial generation. Finally, the N improved programs 555 generated can be evaluated against a hidden unit test 536.

[0055] In some situations, generating a program to solve a problem, especially a competition-level programming problem, requires a vast search space of possible programs. Very often, all programs fail completely, i.e., all programs fail against exemplary tests.

number

[0056] In one embodiment, the same critical model (φ) used in program improvement procedure 550 is also used. test ) used in module 561 failure set

number

number

[0057] In one embodiment, the program repair model 566 is designed as a sequence-to-sequence generation model. The input sequence is problem description D 505 and the buggy program W fail This is a linkage. Additional signals received from the unit test result 112 include the type of test outcome, and may also be included in the input sequence, for example, CompileError, RuntimeError, FailedTest, PassedTest, and one of the error subtypes (e.g., syntax error, out-of-index error, and / or similar). The error type is extracted from the error trace returned by the compiler.

[0058] To train program repair model 566, synthetic sample 133, originally used in RL training to train actor-critic network 150, was used for program W due to a bug. fail =W s It is used as such. The ground truth program W106 can be used as the expected correct program. The training objective of the program repair model is the cross-entropy loss, i.e.,

number

[0059] In one embodiment, program 533 may be generated in mini-batches to improve efficiency during inference, and nuclear sampling with a batch size of N=200 may be used. Note that additional computational costs may be incurred to resample using seed sequence 545 during program refinement, but only partial programs need to be generated in the regeneration stage. In this way, the program refinement stage may be less expensive than conventional program synthesis.

[0060] [Computer Environment] Figure 6 is a simplified diagram of a computing device 600 for implementing the reinforcement learning-based program synthesis framework shown in Figures 1 to 5, according to several embodiments. As shown in Figure 6, the computing device 600 includes a processor 610 coupled to memory 620. The operation of the computing device 600 is controlled by the processor 610. Although the computing device 600 is illustrated as having only one processor 610, it is understood that the processor 610 may represent one or more central processing units, multicore processors, microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), etc., within the computing device 600. The computing device 600 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0061] Memory 620 may be used to store software run by computing device 600 and / or one or more data structures used during the operation of computing device 600. Memory 620 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips, or cartridges, and / or any other media adapted for a processor or computer to read from there.

[0062] The processor 610 and / or memory 620 can be located in any suitable physical arrangement. In some embodiments, the processor 610 and / or memory 620 may be mounted on the same board, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), etc. In some embodiments, the processor 610 and / or memory 620 may include distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 610 and / or memory 620 may be located in one or more data centers and / or cloud computing facilities.

[0063] In some examples, memory 620 may include a non-temporary, tangible, machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 610), can cause one or more processors to perform methods further described herein. For example, as illustrated, memory 620 includes instructions for a program synthesis module 630 that can be used to implement and / or emulate systems and models and / or to implement any of the methods further described herein. The program synthesis module 630 may receive input 640 containing a natural language problem specification via a data interface 615 and generate a code program as output 650.

[0064] In some embodiments, the program synthesis model 630 includes an actor network module 631 (similar to 130 in Figure 1), a critic network module 632 (similar to 140 in Figure 1), and a language model 633 (similar to 110 or 120 in Figure 1). Details of the program synthesis module 630 and its submodules 631-633, as well as their interactions, will be described with reference to Figures 1-5.

[0065] In one embodiment, the program synthesis module 630 and its submodules 631 to 633 may be implemented by hardware, software, and / or a combination thereof.

[0066] Some examples of computing devices, such as computing device 400, may include non-temporary, tangible, machine-readable media containing executable code that, when executed by one or more processors (e.g., processor 410), causes one or more processors to execute the process of the Method. Some common forms of machine-readable media that may contain the process of the Method include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0067] Figure 7 is a simplified block diagram of a networked system suitable for implementing the program synthesis framework described in Figures 1 to 5 and other embodiments described herein. In one embodiment, block diagram 700 shows a system including a user device 710 which can be operated by a user 740, data vendor servers 745, 770, and 780, server 730, and other forms of devices, servers, and / or software components that operate to perform various methodologies according to the embodiments described. Exemplary devices and servers may include devices, standalone, and enterprise-class servers, similar to the computing device 100 described in Figure 1, and run an OS such as MICROSOFT® OS, UNIX® OS, LINUX® OS, or other suitable device and / or server-based OS. It should be understood that the devices and / or servers shown in Figure 7 may be deployed and arranged in other ways, and the operations performed by such devices and / or servers, and / or services provided, may be combined or separated for a given embodiment, and may be performed by more or fewer devices and / or servers. One or more devices and / or servers may be operated and / or maintained by the same or different entities.

[0068] The user device 710, data vendor servers 745, 770, and 780, and server 730 may communicate with each other via network 760. The user device 710 may be utilized by a user 740 (e.g., a driver, system administrator, etc.) to access various features available to the user device 710, which may include processes and / or applications associated with server 730 for receiving output data anomaly reports.

[0069] The user device 710, the data vendor server 745, and the server 730 may each include one or more processors, memories, and other suitable components for executing instructions such as program code and / or data stored on one or more computer-readable media to implement the various applications, data, and steps described herein. For example, such instructions may be stored on one or more computer-readable media, such as memory or data storage devices, that are internal to and / or external to the various components of the system 700 and / or accessible via the network 760.

[0070] The user device 710 may be implemented as a communication device that can utilize appropriate hardware and software configured for wired and / or wireless communication with the data vendor server 745 and / or server 730. For example, in one embodiment, the user device 710 may be implemented as other types of computing devices capable of transmitting and / or receiving data, such as an autonomous vehicle, a personal computer (PC), a smartphone, a laptop / tablet computer, a wristwatch with appropriate computer hardware resources, eyeglasses with appropriate computer hardware (e.g., GOOGLE GLASS®), other types of wearable computing devices, an embedded communication device, and / or an iPad® made by APPLE®. Although only one communication device is shown, multiple communication devices may function similarly.

[0071] The user device 710 in Figure 7 includes a user interface (UI) application 712 and / or other applications 716, which may correspond to executable processes, procedures, and / or applications having associated hardware. For example, the user device 710 may receive a message from the server 730 indicating a generated program and display the message via the UI application 712. In other embodiments, the user device 710 may include additional or different modules having dedicated hardware and / or software, as needed.

[0072] In various embodiments, the user device 710 includes other applications 716, which may be desired in certain embodiments to provide functionality to the user device 710. For example, other applications 716 may include security applications for implementing client-side security features, programmatic client applications for interfacing with appropriate application programming interfaces (APIs) over the network 760, or other types of applications. Other applications 716 may also include communication applications such as email, text, voice, social networking, and IM applications that enable the user to send and receive emails, phone calls, texts, and other notifications over the network 760. For example, other applications 716 may be email or instant messaging applications that receive predictive result messages from the server 730. Other applications 716 may include device interfaces and other display modules that can receive input and / or output information. For example, other applications 716 may include processor-executable software programs for asset management, including a graphical user interface (GUI) configured to provide the user 740 with an interface for viewing the generated programs.

[0073] The user device 710 may further include a database 718 stored in the temporary and / or non-temporary memory of the user device 710, the database 210 which stores various applications and data and may be used during the execution of various modules of the user device 710. The database 718 may store a user profile of user 740, predictions previously viewed or saved by user 740, historical data received from server 730, and / or similar. In some embodiments, the database 718 may be local to the user device 710. However, in other embodiments, the database 718 may be external to the user device 710 and accessible by the user device 710, including a cloud storage system and / or database accessible via network 760.

[0074] The user device 710 includes at least one network interface component 719 adapted to communicate with the data vendor server 745 and / or server 730. In various embodiments, the network interface component 719 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet® device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency, infrared, Bluetooth®, and short-range communication devices.

[0075] The data vendor server 745 may correspond to a server that hosts one or more of the databases 703 a-n (or collectively referred to as 703) and provides the server 730 with training datasets containing publicly available code data. Database 703 may be implemented by one or more relational databases, distributed databases, cloud databases, and / or similar.

[0076] The data vendor server 745 includes at least one network interface component 726 adapted to communicate with user devices 710 and / or server 730. In various embodiments, the network interface component 726 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet® device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency, infrared, Bluetooth®, and short-range communication devices. For example, in one embodiment, the data vendor server 745 may transmit asset information from database 703 to server 730 via the network interface 726.

[0077] The server 730 may house the program synthesis module 630 and its submodules as shown in Figure 1. In some implementations, the module 630 may receive training code data from the database 719 in the data vendor server 745 via the network 760 to generate a program. The generated program may be sent via the network 760 to a user device 710 for review by the user 740.

[0078] The database 732 may be stored in the temporary and / or non-temporary memory of the server 730. In one embodiment, the database 732 may store data obtained from the data vendor server 745. In one embodiment, the database 732 may store parameters of the program synthesis model 630. In one embodiment, the database 732 may store previously generated programs and problem descriptions, as well as corresponding input feature vectors.

[0079] In some embodiments, the database 732 may be local to the server 730. However, in other embodiments, the database 732 may be external to the server 730 and accessible by the server 730, including a cloud storage system and / or database accessible via the network 760.

[0080] Server 730 includes at least one network interface component 733 adapted to communicate with user devices 710 and / or data vendor servers 745, 770, or 780 via network 760. In various embodiments, the network interface component 733 may comprise a variety of other types of wired and / or wireless network communication devices, including DSL (e.g., digital subscriber line) modems, PSTN (public switched telephone network) modems, Ethernet® devices, broadband devices, satellite devices, and / or microwave, radio frequency (RF), and infrared (IR) communication devices.

[0081] Network 760 may be implemented as a single network or a combination of multiple networks. For example, in various embodiments, network 760 may include the Internet, one or more intranets, a terrestrial communication line network, a wireless network, and / or other suitable types of networks. Thus, network 760 can correspond to small-scale communication networks, such as private or local area networks, or larger networks, such as wide area networks or the Internet, accessible by various components of system 700.

[0082] [Example Workflow] Figure 8 is an exemplary logic flowchart illustrating a reinforcement learning-based training method for program synthesis based on the actor-critic framework shown in Figure 1, according to some embodiments described herein. One or more of the processes of Method 800 may be implemented in the form of executable code stored on a non-temporary, tangible, machine-readable medium, which, when executed by one or more processors, can cause one or more processors to perform one or more of the processes. In some embodiments, Method 800 corresponds to the operation of a program synthesis module 630 (for example, Figures 6-7).

[0083] In step 802, the problem specification (e.g., 105 in Figure 1) and the corresponding solution program (e.g., 106 in Figure 1) may be received via an input interface (e.g., 615 in Figure 6, 733 in Figure 7).

[0084] In step 804, the pre-trained language model (e.g., 120 in Figure 1) may be fine-tuned based on the problem specification and the corresponding solution program.

[0085] In step 806, a finely tuned pre-trained language model (e.g., 130 in Figure 1) may, in the decoding time step, generate a sampled program (e.g., 133 in Figure 1) in response to the problem specification (e.g., 105 in Figure 1). For example, in one embodiment, the predicted token w of the sampled program t The result can be generated governed by the current parameters of a finely tuned pre-trained language model (e.g., 130 in Figure 1) at the decoding timestep t. The hidden state representation of the finely tuned pre-trained language model may be updated accordingly to represent the next predicted token w of the sampled program. t+1 This may be generated in the next decoding timestep t+1 using an updated hidden state representation of a finely tuned pre-trained language model.

[0086] In step 808, the critical model (e.g., 140 in Figure 1) may generate a return indicating the functional accuracy of the sampled program based on a comparison of the execution result of the sampled program with the test results of the problem specification. For example, in one embodiment, a return is generated when an exit token is generated for the sampled program. The sampled program (e.g., 133 in Figure 1) and the test (e.g., 112 in Figure 1) may be passed to the compiler, and the first return value may be determined, for example, according to equation (2), depending on whether the sampled program was successfully compiled and executed and whether the execution result matches the test results of the problem specification.

[0087] In step 810, the policy gradient (e.g., the gradient of 135 in Figure 1) of the expected value of the return given the current parameters of the finely tuned pre-trained language model can be calculated. For example, in one embodiment, the policy gradient is calculated as an estimate based on the return and gradient of the conditional probabilities of the previously predicted tokens and the predicted tokens conditioned by the problem specification, for example, according to equation (4).

[0088] In one implementation, the policy gradient can be calculated using a baseline comparison. For example, a baseline program generated by a base model in response to a problem specification can be input to a critical model (e.g., 140 in Figure 1). A second return value can be determined depending on whether the baseline program was successfully compiled and executed, and whether the execution result matches the test result of the problem specification. In this way, the policy gradient is calculated, for example, according to equation (5), based on the difference between the first and second return values, as well as the gradient of the conditional probabilities of the previous prediction token and the prediction token conditioned by the problem specification.

[0089] In other embodiments, the policy gradient is calculated, for example, according to equation (9), based on the probability distribution of the predicted test outcomes generated by the critical model and the gradient of the conditional probabilities of the previously predicted tokens and the predicted tokens conditioned by the problem specification.

[0090] In step 812, the fine-tuned pre-trained language model (e.g., 130 in Figure 1) may be updated according to the policy gradient.

[0091] In one implementation, a critic model (e.g., 140 in Figure 1) is trained. The critic model receives a training sequence of the problem specification (e.g., 105 in Figure 1) and the sampled program (e.g., 133 in Figure 1) and can generate a predicted test outcome corresponding to the sampled program. The predicted test outcome is calculated, for example, by a softmax operation of the maximum pooled context hidden state of the decoder in the critic model, according to equations (6) to (7). The cross-entropy loss is calculated by comparing the predicted test outcome with the execution result of the sampled program, for example, according to equation (8), and the critic model may be updated based on the cross-entropy loss.

[0092] Figure 9 is an exemplary logic flow diagram illustrating a program synthesis method based on the LM shown in Figure 5, according to some embodiments described herein. One or more of the processes of Method 900 may be implemented, at least partially, in the form of executable code stored on a non-temporary, tangible, machine-readable medium, which, when executed by one or more processors, can cause one or more processors to perform one or more of the processes. In some embodiments, Method 800 corresponds to the operation of a program synthesis module 630 (for example, Figures 6-7).

[0093] In step 902, the problem specification (e.g., 505 in Figure 5) may be received by a pre-trained language model for program synthesis (e.g., 130 in Figure 5) via an input interface (e.g., 615 in Figure 6, 733 in Figure 7).

[0094] In step 904, one or more unit test input-output pairs (e.g., 112 in Figure 2) may be extracted from the problem specification (e.g., 105 in Figure 2).

[0095] In step 906, the language model may generate several program samples (for example, 533 in Figure 5) from the problem specification.

[0096] In step 908, one or more unit tests (e.g., 112 in Figure 5) may be applied to multiple program samples (e.g., 533 in Figure 5) based on one or more unit test input-output pairs.

[0097] In step 910, from multiple program samples, a first set of program samples that passed one or more unit tests (e.g., 541 in Figure 5) and a second set of program samples that failed (e.g., 542 in Figure 5) may be determined. For example, the program samples in the second set include at least one of the following: compilation errors, runtime errors, and failure to pass at least one of the unit tests.

[0098] In step 912, the Critic Model can determine the value for the second program sample in the second set based on the predicted probability that the second program sample will pass one or more unit tests, for example, according to equation (11).

[0099] In step 914, a subset of program samples with the highest value from the second set (e.g., 565 in Figure 5) may be selected. The input sequence (e.g., 566) is formed by concatenating the problem specification, the selected program samples, and error information corresponding to the selected program samples. For example, the error information may include the unit test outcome corresponding to the selected program samples and either a compilation or runtime error subtype of the selected program samples.

[0100] In step 916, the program repair model may be used to generate repaired program samples based on the input sequence. For example, the program repair model compares program samples that have failed unit tests with ground truth programs corresponding to the problem specification and is trained by training objectives conditioned on the unit test outcomes and / or error subtypes corresponding to the program samples.

[0101] In step 918, one or more subsequences (e.g., 543 in Figure 5) may be selected from a first set of program samples via critical scoring. Each subsequence is a truncated version of a program sample. For example, in one embodiment, a critical model may determine a value for each token in the first set of program samples based on the predicted probability that a subsequence up to each token will pass one or more unit tests, according to equation (10), for example. A particular token in the first program sample with the highest value is identified, and a subsequence of the first program sample up to that particular token may be selected as a subsequence. If the selected subsequence contains a particular token up to that particular token for which the probability of the corresponding subsequence failing one or more unit tests is higher than the probability of it passing, the selected subsequence is further truncated at that particular token.

[0102] In step 920, the language model may, for example, use a subsequence as a "seed" (e.g., 545 in Figure 5) to generate the remaining tokens conditioned by one or more subsequences.

[0103] In step 922, the remaining tokens generated from step 920 can be combined with one or more subsequences to generate one or more improved program samples.

[0104] [Example Data Experiment] In the exemplary data experiments of the proposed RL-based program synthesis framework shown in Figures 1-5 and the workflow in Figures 8-9, the CodeT5-large model (770M) is pre-trained from scratch according to the T5-large architecture. In particular, the code-specific tokenizer is taken from the CodeT5 research, and since the C / C# dataset is not publicly available, six programming languages ​​(PLs) from CodeSearchNet (described in Husain et al., Codesearchnet challenge: Evaluating the state of semantic code search, Computing Research Repository (CoRR), abs / 1909.09436, 2019) (CSN) are used instead of the eight PLs in CodeT5. Only the pre-training task of masked span prediction (MSP) is applied, and therefore the model does not need to parse the program into an abstract syntax tree (AST) to obtain identifier information.

[0105] The final preprocessing step was required in other original pre-training tasks, such as masked identifier prediction in the original CodeT5 study. To further speed up training, the data samples were concatenated into batch size 512 for pre-training with MSP, resulting in a number of tokens of 1.1B. To validate the advantages of using this new pre-trained CodeT5 as the base model (e.g., 130 in Figure 1), the model is evaluated in CodeXGLUE.

[0106] The exemplary data experiment was run on Kubernetes using 16 A100-40G GPUs on Google Cloud Platform, with a total pre-training period of approximately 21 days. The first pre-training stage using MSP employed a 15% corruption rate, a peak learning rate (LR) of 2e-4, and a batch size of 2048. CSN was pre-trained for 150 epochs (10 days), followed by 10 epochs (5 days) with GCPY. For the second stage of pre-training using NTP, a peak LR of 1e-4 and a batch size of 256, along with 10 epochs (6 days) of pre-training, were employed. The maximum lengths for this purpose were set to 768 and 600 for the source and target sequences, respectively. For all experiments, the AdamW optimizer with a weight decay of 0:05 and a linear decay LR scheduler with 1000 warm-up steps were employed.

[0107] The models are evaluated using the pass@k metric, which is the percentage of problems solved using k generated programs per problem, according to (Hendrycks et al., Measuring coding challenge competence with apps, in proceedings of NeurIPS, 2021; Chen et al., Evaluating large language models trained on code, arXiv preprint, arXiv:2107.03374, 2021). The n@k metric is used according to (Li et al., Competition-level code generation with alphacode, arXiv preprint, arXiv:2203.07814, 2022), which considers only a subset of n candidates from the k generated programs per problem. The subset of n candidates is typically selected by a filtering method that involves passing the generated programs through exemplary tests given as part of the problem description.

[0108] An exemplary benchmark for comparison is the APPS Program Synthesis Benchmark (see Hendrycks et al.), because it contains a large number of coding problems of varying difficulty collected from multiple coding websites. APPS consists of 10,000 coding problems with 50-50 training / test splits. Each problem has an average of 23.2 correct Python programs and 21.2 unit tests. The average length per problem is 293.2 words, and the average length per program is 18.0 lines. The dataset is categorized into three difficulty levels: Introductory (3639, training / test = 2639 / 1000), Interview (5000, training / test = 2000 / 3000), and Competition (1361, training / test = 361 / 1000). Each sample includes an average of 20 unit tests to verify the functional accuracy of the program. The same preprocessing step described by Hendrycks et al. is used to formulate the input sequence from the problem description.

[0109] On APPS, the pre-trained CodeT5 is fine-tuned using the RL-based framework shown in Figure 1. To warm-start the CodeT5 model with Lce, a batch size of 64 and a warm-up LR from 0 to 2e-5 are used over the first 500 steps, decaying polynomially (power = 0:5) to 1e-5 until the end of 10 epochs, which takes approximately 30 hours on a single A100 GPU. The maximum source and target sequence lengths are set to 600 and 512, respectively.

[0110] An additional benchmark is the MBPP [Mostly Basic Programming Problems] benchmark, a smaller and simpler Python program synthesis dataset (described in Austin et al., Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021) for evaluation. The dataset contains 974 instances, with 374 / 90 / 500 instances for training / validation / testing respectively, and 10 instances reserved for few-shot learning. The problems are typically short, usually each being a single sentence in natural language description. Each problem is accompanied by one correct solution (averaging 6.8 lines of code) and three unit tests in the form of assert statements to verify functional correctness. Unlike APPS, the unit tests in MBPP are not hidden but are explicitly built into the source sequence for the program synthesis model. This could, very rarely, encourage the model to overfit to these assert statements through hardcoding if expressions. However, for a fair comparison with the baseline, the source sequence is constructed in the same way as in previous studies. Specifically, using the same prompt format as Austin et al., the input sequence is prepared as follows: Problem description + "Your code should satisfy these tests:" + 3 assert statements.

[0111] On MBPP, experiments are conducted in both zero-shot (Section 4.5) and full fine-tuning setups. To fine-tune CodeT5, because the MBPP training set is small, the model is fine-tuned over 60 epochs with a constant LR of 2e-5 and a batch size of 32, which takes less than 30 minutes on one A100. The maximum source and target lengths are set to 382 and 306, respectively.

[0112] Exemplary baselines include GPT2 (Radford et al., Language models are unsupervised multitask learners, OpenAI blog, 1(8):9, 2019), GPT-Neo (Black et al., GPT-NEO: Large scale autoregressive language modeling with mesh-tensorflow. URL https: / / doi.org / 10.5281 / zenodo, 5297715, 2021), and GPT3 (Brown et al., Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020) for comparison with the RL-based framework described herein (referred to as “CodeRL”). The results are also compared with Codex (see Chen et al.) and AlphaCode (see Li et al.). Note that, by default, the results for pre-trained LMs (excluding Codex and GPT3) are from models fine-tuned on APPS using only the standard loss Lce. CodeRL is model-independent and can therefore be integrated with GPT variants such as GPT-J and GPT-Neo.

[0113] Figure 10(a) shows that CodeRL with the CodeT5 model can achieve a significant performance gain, outperforming many pre-trained LMs of much larger size. Specifically, CodeRL achieved new state-of-the-art results of 2:69% pass@1, 6:81% pass@5, and 20:98% pass@1000. Figure 10(b) shows that when evaluated on a subset of filtered code samples, CodeRL+CodeT5 can achieve state-of-the-art results of 8:48% 1@k and 12:62% 5@k.

[0114] Secondly, the advantages of upsampling generation are seen when increasing the number of generated samples k from 1 to 1000. Note that while CodeRL incurs additional computational costs during inference in CS, CodeRL requires only a much lower k to achieve comparable performance to other models.

[0115] Specifically, with k=1000 alone, CodeRL performance is as good as AlphaCode, which has a much larger generation budget of k=50000. Finally, Figure 10(b) also shows that fine-tuning can significantly improve model performance for challenging programming tasks at the interview and competition levels. Specifically, Codex, which is not fine-tuned on APPS and tested with few-shot settings, can achieve good n@1000 results, but the model dramatically fails on synthesis tasks at the interview and competition levels. This observation highlights a considerable gap between the pre-training stage and downstream synthesis tasks.

[0116] Figure 11 shows the results of CodeT5-770M trained using different approaches to estimate the return of code samples. Overall, the CodeRL objective with relative token-level return estimates by the Critic model (Model D) can achieve the best performance in pass@1 and pass@5. Secondly, it should be noted that using absolute returns without a baseline (Model B) can result in the greatest performance degradation because this approach imposes a heavy penalty on all inaccurate samples (even if they are still better than a naive baseline). Therefore, considering relative return estimates that can effectively utilize incomplete code can result in a better synthetic system. Thirdly, in the absence of a Critic model, simply assigning the same reward to all tokens in a code sample (Model A) is disadvantageous because these return estimates are too restrictive to be used as feedback signals for RL training. For example, a program can be considered inaccurate only by an additional whitespace character, which can result in an indentation error in a Python program. Simply assigning the same reward to all tokens in this program would impose a large penalty on the correct part of the program sequence. Finally, token values

number

[0117] Figure 12 shows the results for different combinations of Lce and Lrl. Since CodeRL is model-independent, experiments are performed on both CodeT5 and GPT-Neo. Note that in these experiments, Lce and Lrl are applied to a model that has already been warm-started / fine-tuned with Lce for up to 10 epochs. Firstly, when using Lrl alone, a problem was observed where the gradient vanished during fine-tuning. Therefore, the final model actually deteriorates, leading to a decrease in performance. Secondly, by using Lce alone for further fine-tuning, the model performance actually deteriorates during test time, despite the improvement in loss during training time. Therefore, these models are expected to overfit to the training data, as is similarly seen in the analysis of the pre-trained models in Figure 16.

[0118] In addition, Lce's naive approach using synthetic samples Ws (all of which are treated as correct code with r(Ws)=1) is still observed to yield some performance improvement with GPT-Neo in pass@5. However, in all other cases, this training strategy does not work as well as considering a critical model to estimate the returns of Ws by their test results. Finally, it is observed that using both Lce and Lrl yields more consistent overall performance improvements in pass@1 and pass@5 for both the GPT-Neo and CodeT5 models.

[0119] Figure 13 shows the ablation results of critical sampling (CS) during inference applied to the CodeT5 model. Different combinations of program improvement and repair steps are tested. Overall, across all metrics, a positive effect of CS combining both program improvement and repair is seen, with a more significant gain, particularly at pass@1000. Note that program improvement alone also helps improve performance, but its effect is reduced for the 1@1000 metric. Note that n@k measures the resolution rate among a filtered subset P of k samples. Since program improvement technically increases the size of this subset, the n@k metric considers an exponentially larger number of n sample options than before. This normalizes n@k by a larger pool of n candidate sets, making the impact of program improvement on model performance less significant.

[0120] Secondly, (for the problem in the case of P=;) when program improvement is integrated with program repair, further performance is obtained across all metrics. Interestingly, when experimenting with different higher M selection schemes, the best overall performance is found at M=1, and performance begins to decline from M=2 to M=4 (except for the pass@200 result). This observation demonstrates the advantage of using a critical model rather than selecting multiple program candidates to focus on the best candidate for program repair. Furthermore, at larger M, each program candidate will have a smaller batch size (i.e., N=M). This makes it less likely that the program repair model will properly repair and generate the correct program.

[0121] In one embodiment, the data experiment examines a subset of APPS test splits containing the highest difficulty level test samples (i.e., competition programming tasks). Figure 14 shows the pass@k and n@k results for CodeRL+CodeT5 and CodeT5 only, with k ranging from 1 to 200 and n=f1.5g. Since CodeRL is model-independent, it is integrated with GPT-J and the results are reported. To focus on the impact of RL optimization, during test time, the model is compared to one using only kernel sampling and without the CS procedure.

[0122] Figure 14 shows that the performance gains are quite consistent for both GPT-J and CodeT5. In particular, as k increases, the performance gain of CodeRL becomes more pronounced in both the GPT-J and CodeT5 models. These gains are thought to be due to the CodeRL learning objective Lrl, which prompts the model to explore code solutions derived from the model's sampling distribution. During test time, as the k sampling budget increases, the model becomes capable of generating a wider variety of code solutions, and the effect of Lrl becomes more pronounced.

[0123] In one embodiment, the performance of the synthesis system is correlated with the quality of the underlying model. Figure 15 reports the results of CodeT5 using different configurations of model size, pre-training data, and pre-training objectives. For fair comparison, all models are only fine-tuned / warm-started on APPS with Lce up to 12 epochs. It can be seen that scaling up the number of model parameters (from 60M to 770M) can significantly improve the model performance on downstream synthesis tasks. When the pre-training data is improved by adding the GCPY dataset (10 times larger than the CSN dataset), good performance improvements can be seen, i.e., from 1:3 to 1:56 pass@1, and from 1:72 to 2:06 pass@5. Finally, by combining pre-training objectives from Masked Span Prediction (MSP) and Next Token Prediction (NTP), the model can be better adapted to downstream synthesis.

[0124] Figure 16 shows the performance of CodeT5 model variants by fine-tuning the epoch and the difficulty level of the programming task. Note that in these experiments, the data experiments only compare CodeT5 model variants by pre-training strategy and therefore only involve Lce in the fine-tuning stage on APPS. Consistent with our previous analysis, strengthening both the pre-training data (with larger data from GCPY) and the pre-training objective (with the NTP objective) generally improves model performance over training epochs. Furthermore, as shown by the analysis of the training objective, using only Lce often results in overfitting performance, typically after epoch 10 in our case. Therefore, to further fine-tune large-scale LMs, it is beneficial to utilize synthetic training samples and employ the RL objective Lrl to avoid overfitting the model.

[0125] Figure 17 reports the results of our CodeRL+CodeT5 against the MBPP benchmark compared to a finely tuned GPT model up to size 137B. CodeRL+CodeT5(ZS) was trained on APPS and then evaluated on MBPP in a zero-shot setting. CodeRL with the very small model size CodeT5 was found to deliver surprisingly good zero-shot performance, setting a new state-of-the-art result of 63.0% pass@80 compared to 61.4% pass@80 for GPT-137B. This validates CodeRL's strong zero-shot transfer capability for invisible tasks.

[0126] A common concern regarding transfer learning is that the source (APPS) task and the target (MBPP) task may have overlapping training data, and consequently, the source model may tend to remember this substantially similar data when applied to the target task. To address this concern, following Austin et al., we analyze how many lines of code appear in both the APPS training set and the MBPP program. For this analysis, code comments are discarded, whitespace is normalized for each line, and then lines that appear more than twice anywhere in MBPP are excluded, as they are likely to be common Python keywords such as return and break.

[0127] Figure 18 shows the absolute number of overlapping rows (left) and the relative fraction of overlapping rows (right) in MBPP programs. As can be seen from the figure, the overlap between APPS and MBPP appears to be minimal. Only 12.6% of MBPP programs have more than half of their rows matching somewhere in the APPS training data. Furthermore, more than half of the programs (514 out of 974) have zero overlap, and 90.9% have three or fewer overlapping rows with the APPS training set. Moreover, if contiguous lines are required, there are two or fewer consecutive overlapping rows.

[0128] Figure 19 shows the average percentage of generated programs per problem, grouped by their test outcomes. Specifically, CodeT5 or CodeRL+CodeT5 is used to generate programs, and 200 generated programs are randomly selected per test sample in the APPS test split. The programs are passed to either exemplary or hidden unit tests, and the output programs are grouped by their test outcomes. The outcomes are categorized according to the definition of Equation (2), which includes CompileError, RuntimeError, FailedTest, and PassedTest.

[0129] Firstly, integrating CodeRL into both exemplary and implicit unit tests can increase the likelihood that a program will pass tests and decrease the probability of one or more unit tests failing. The probability of passing unit tests is improved more significantly in introductory-level programming problems.

[0130] Secondly, it is noteworthy that the percentage of programs created with CodeRL that have compilation errors decreases, leading to a greater impact on interview and competition-level problems. Because CodeRL programs have a low probability of having compilation errors, these programs still suffer from runtime errors. This, in turn, increases the probability that CodeRL programs contain runtime errors.

[0131] Note that there is a fairly significant performance gap in the test outcomes between the exemplary unit tests (Figure 19(a)) and the hidden unit tests (Figure 19(b)). This observation suggests that the exemplary tests are not as comprehensive as the hidden tests and therefore limit the positive impact of the CodeRL generation procedure due to false positives.

[0132] Figure 20 shows a programming problem from the APPS benchmark and an example of the corresponding program generated by a CodeT5 variant. Specifically, a CodeT5 model fine-tuned only by Lce, based on the same foundational pre-trained CodeT5 (pre-trained for GCPY data and NTP purposes), is compared to other models following the CodeRL framework. CodeRL+CodeT5 shows the program before and after applying the CS procedure. It can be seen that applying CodeRL can generate a more appropriate program, and that using the CS procedure further improves their functional accuracy. For example, in Figure 20, the CodeT5 model misunderstands the problem and focuses only on finding the greatest common divisor between a and b. Instead, the CodeRL model avoids this error and tackles the problem of finding the greatest common divisor between the factorials of a and b.

[0133] Furthermore, CodeRL has been shown to improve the complexity of generated programs, a key characteristic in complex programming problems. For example, in the interview-level program in Figure 20, without applying CS, the generated program is functionally correct but fails during execution due to a timeout error. This program simply calculates the separate factorials of both a and b, which slows down execution in scenarios with extremely large a or b. Applying the CS procedure can condition the model of previous parts of the program and (re)generate new tokens to produce a more efficient program. In the example in Figure 20, we calculate the factorial for min(a,b) to improve the program's efficiency. Thus, the resulting final program can pass all hidden unit tests (including those with extremely large values) without timeout errors.

[0134] This description and accompanying drawings illustrating aspects, embodiments, implementations, or applications of the invention should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational modifications can be made without departing from the spirit and scope of this specification and the claims. In some examples, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure the embodiments of this disclosure. Similar numbers in two or more figures represent the same or similar elements.

[0135] This description provides specific details illustrating several embodiments consistent with the present disclosure. Numerous specific details are provided to provide a complete understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments can be carried out without some or all of these specific details. The specific embodiments disclosed herein are illustrative but not limiting. Those skilled in the art will recognize other elements not specifically described herein but within the scope and spirit of the disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in relation to one embodiment may be incorporated into other embodiments unless otherwise specifically described, or if one or more features would render the embodiment non-functional.

[0136] While exemplary embodiments have been shown and described, a wide range of modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be adopted without corresponding use of other features. Those skilled in the art will recognize many variations, substitutions, and modifications. Therefore, the scope of the present invention should be limited only by the following claims, which should be interpreted to broadly coincide with the scope of the embodiments disclosed herein.< / endoftext> < / endoftext>

Claims

1. A method for a processor to perform reinforcement learning-based training for program synthesis based on a reinforcement learning framework, The input interface receives the problem specification and the corresponding solution program, Based on the aforementioned problem specification and the corresponding solution program, the pre-trained language model is fine-tuned, The finely tuned pre-trained language model generates a sampled program in response to the problem specification during the decoding time step. The critical model generates a return indicating the functional accuracy of the sampled program based on a comparison of the execution results of the sampled program with the test results of the problem specification, Given the current parameters of the finely tuned pre-trained language model, the policy gradient of the expected value of the return is calculated. Updating the finely tuned pre-trained language model according to the aforementioned policy gradient, Methods that include...

2. The sampled program, In the decoding time step, predictive tokens for the sampled program are generated, which are governed by the current parameters of the finely tuned pre-trained language model. Updating the hidden state representation of the finely tuned pre-trained language model, In the next decoding time step, the updated hidden state representation of the fine-tuned pre-trained language model is used to generate the next predicted token of the sampled program, and the tokens generated are: The method according to claim 1.

3. The method according to claim 2, wherein the return is generated when an exit token is generated for the sampled program.

4. The aforementioned return is, Passing the sampled program and tests to the compiler, The first return value is determined depending on whether the sampled program was successfully compiled and executed, and whether the execution result matches the test result of the problem specification. The method according to claim 1, which is produced by

5. The method according to claim 4, wherein the policy gradient is calculated as an estimate based on the return and the gradient of the conditional probabilities of the previous prediction token and the prediction token conditioned by the problem specification.

6. Inputting the baseline program generated by the base model in response to the aforementioned problem specification into the critic model, The second return value is determined depending on whether the baseline program was successfully compiled and executed, and whether the execution result matches the test result of the problem specification. The method according to claim 4, further comprising:

7. The method according to claim 6, wherein the policy gradient is calculated based on the difference between the first return value and the second return value and the gradient of the conditional probabilities of the previous prediction token and the prediction token conditioned by the problem specification.

8. The aforementioned Critic Model, The Critic Model receives the training sequence of the problem specification and the sampled program, The aforementioned critical model generates predicted test outcomes corresponding to the sampled programs, The cross-entropy loss is calculated by comparing the predicted test outcome with the execution results of the sampled program, Updating the Critic model based on the cross-entropy loss, The method according to claim 1, which is trained by

9. The method according to claim 8, wherein the predicted test outcome is calculated by a softmax operation of the maximum pooled context hidden state of the decoder in the critical model.

10. The method according to claim 9, wherein the policy gradient is calculated based on the probability distribution of the predicted test outcome generated by the critical model and the gradient of the conditional probabilities of the predicted tokens, conditioned by the previously predicted tokens and the problem specification.

11. A reinforcement learning-based training system for program synthesis based on a reinforcement learning framework, An input interface that receives a problem specification in a language model pre-trained for program synthesis, Memory that stores multiple processor-executable instructions, The aforementioned plurality of processor-executable instructions are read and executed, The input interface receives the problem specification and the corresponding solution program, Based on the aforementioned problem specification and the corresponding solution program, the pre-trained language model is fine-tuned, The finely tuned pre-trained language model generates a sampled program in response to the problem specification during the decoding time step. The critical model generates a return indicating the functional accuracy of the sampled program based on a comparison of the execution results of the sampled program with the test results of the problem specification, Given the current parameters of the finely tuned pre-trained language model, the policy gradient of the expected value of the return is calculated. Updating the finely tuned pre-trained language model according to the aforementioned policy gradient, A processor that performs operations including, A system that includes this.

12. The sampled program is In the decoding time step, predictive tokens for the sampled program are generated, which are governed by the current parameters of the finely tuned pre-trained language model. Updating the hidden state representation of the finely tuned pre-trained language model, In the next decoding time step, the updated hidden state representation of the fine-tuned pre-trained language model is used to generate the next predicted token of the sampled program, and the tokens generated are: The system according to claim 11.

13. The system according to claim 12, wherein the return is generated when an end token is generated for the sampled program.

14. The return is, Passing the sampled program and tests to the compiler, The first return value is determined depending on whether the sampled program was successfully compiled and executed, and whether the execution result matches the test result of the problem specification. The system according to claim 11, which is generated by...

15. The system according to claim 14, wherein the policy gradient is calculated as an estimate based on the return and the gradient of the conditional probabilities of the previous prediction token and the prediction token conditioned by the problem specification.

16. Inputting a baseline program generated by the base model in response to the problem specification into the critic model, The second return value is determined depending on whether the baseline program was successfully compiled and executed, and whether the execution result matches the test result of the problem specification. The system according to claim 14, further comprising:

17. The system according to claim 16, wherein the policy gradient is calculated based on the difference between the first return value and the second return value and the gradient of the conditional probabilities of the previous prediction token and the prediction token conditioned by the problem specification.

18. The critic model is The Critic Model receives the training sequence of the problem specification and the sampled program, The aforementioned critical model generates predicted test outcomes corresponding to the sampled programs, The cross-entropy loss is calculated by comparing the predicted test outcome with the execution results of the sampled program, Updating the Critic model based on the cross-entropy loss, The system according to claim 11, which is trained by

19. The predicted test outcome is calculated by a softmax operation of the maximum value pooled context hidden state of the decoder in the critical model, The system according to claim 18, wherein the policy gradient is calculated based on the probability distribution of the predicted test outcome generated by the critical model and the gradient of the conditional probabilities of the predicted tokens, conditioned by the previously predicted tokens and the problem specification.

Images