System and method for program synthesis

Abstract

Embodiments described herein provide a reinforcement learning-based framework that joins pre-trained language models (LMs) for program synthesis tasks. In particular, the framework employs a training strategy that optimizes a pre-trained LM for a program synthesis task in an actor-commentator approach.

Description

[0001] Inventors: H. Le, Y. Wang, A. D. Gottemar, C. H. Hoy

[0002] Cross-references

[0003] This international application claims priority to co-pending, commonly owned U.S. non-provisional application Nos. 17 / 896,942 and 17 / 896,946, both filed on August 26, 2022, each of which is a non-provisional application of U.S. provisional application No. 63 / 344,900, filed on May 23, 2022, and claims priority under 35 U.S.C. 119.

[0004] All of the foregoing applications are hereby expressly incorporated herein by reference in their entirety. Technical Field

[0005] Embodiments generally relate to machine learning systems, and more particularly to systems and methods for program synthesis via pre-trained models and deep reinforcement learning. Background Art

[0006] Program synthesis, also commonly referred to as code generation, is the task of generating a computer code program that satisfies a problem specification, e.g., sorting a list, merging two data tables, etc. When program synthesis is viewed as a sequence-to-sequence task, some pre-trained language models can be adapted to receive an input sequence as a problem specification in natural language, and then generate a code sequence as an output program. However, these existing language models may have limited code generation performance because these models typically follow a standard supervised fine-tuning procedure to train program synthesis models only from a natural language problem description and a ground-truth program. Such paradigms largely ignore some important but potentially useful signals in the problem specification, such as unit tests, leading to poor performance when solving complex unknown coding tasks.

[0007] Therefore, an efficient and accurate program synthesis model is needed. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a simplified block diagram illustrating an exemplary architecture employing an actor-critic framework to optimize a pre-trained language model (LM) (and a fine-tuned LM) for program synthesis, according to embodiments described herein.

[0009] Figure 2 is a simplified block diagram illustrating an exemplary program synthesis task according to one embodiment described herein.

[0010] Figure 3is a simplified block diagram illustrating an example of a reinforcement learning-based program synthesis framework for an exemplary program synthesis task, according to one embodiment described herein.

[0011] Figure 4 is an illustration according to the embodiments described herein Figure 1 Simplified block diagram of an exemplary training procedure for a critic network in .

[0012] Figure 5 is an illustration of a method for using a Figure 1 Simplified block diagram of the critic sampling (CS) framework for program synthesis with a trained LM (actor network).

[0013] Figure 6 According to some embodiments, Figures 1 to 5 A simplified diagram of a computing device for a reinforcement learning-based program synthesis framework is shown in FIG.

[0014] Figure 7 is suitable for implementing Figures 1 to 5 A simplified block diagram of a networked system of a program synthesis framework as described in and other embodiments described herein.

[0015] Figure 8 is an illustration based on some embodiments described herein Figure 1 An exemplary logical flow chart of a reinforcement learning-based program synthesis training method for the actor-critic framework shown in FIG.

[0016] Fig. 9 is an illustration based on some embodiments described herein Figure 5 An exemplary logical flow chart of the program synthesis method of LM is shown in FIG.

[0017] Figures 10 to 19 is an illustration according to some embodiments described herein Figures 1 to 9 An exemplary data table or chart of exemplary performance comparisons of the program synthesis framework described in and various baseline models.

[0018] Fig. 20 shows programming problems and Figures 1 to 9 Examples of the program synthesis framework described in and the corresponding programs generated by various baseline models.

[0019] In the drawings, elements with the same reference numerals have the same or similar functions. DETAILED DESCRIPTION

[0020] As used herein, the term "network" may include any hardware or software based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or in conjunction with it.

[0021] As used herein, the term "module" may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0022] Existing language models that can be used for program synthesis are typically trained using a conventional next token prediction (NTP) objective that maximizes the likelihood of the next true token. Training the model with only the next token prediction objective in a "teacher forcing" manner often results in accumulated errors during test time when tokens are generated by conditioning on previously sampled tokens rather than true tokens. This problem becomes even more serious in the field of program synthesis, where existing token matching scores (such as BLEU) may not measure the functional correctness of a complete program.

[0023] Additionally, existing language models may fail to exploit potentially meaningful signals from unit tests, which directly determine model performance through the functional correctness of the program. Current approaches ignore this important signal during model optimization as well as program generation.

[0024] In view of the problems in the existing program synthesis model, the embodiments described herein provide a framework based on reinforcement learning, which is combined with a pre-trained language model (LM) for program synthesis tasks. Specifically, a pre-trained LM (e.g., pre-trained using common code data, etc.) can be fine-tuned for a pair of natural language problem descriptions and corresponding solution programs for program synthesis tasks. Then, the fine-tuned LM can be used as an actor network, which synthesizes sequences generated from actors to form sampled programs that respond to inputs of the same problem description, including correct and incorrect programs. These program samples are passed to a critic model, which is trained as an error predictor to predict the test results of the input program under a given unit test to determine the feedback for evaluating the functional correctness of these program samples. Then, the feedback generated from the critic model is used to calculate the policy gradient to minimize the expected feedback (negative value). Then, the actor network is fine-tuned based on the policy gradient.

[0025] In this way, the pre-trained LM is fine-tuned for the program synthesis task in a reinforcement learning manner. For example, the pre-trained parameters of the LM can serve as a stochastic policy for the actor network, according to which actions can be generated as predictions for each token of the output program. The pre-trained LM (actor network) receives feedback measured by the functional correctness of the generated program, and the goal of reinforcement learning is to minimize the expected reward.

[0026] In one embodiment, during inference, a fine-tuned LM through an RL framework can be used to generate one or more code programs in response to a natural language problem description. To improve the correctness and accuracy of the resulting programs, a programming improvement program and / or a program repair program can optionally be employed to improve and / or repair the generated programs based on the functional correctness of the generated programs during test time. Specifically, an exemplary unit test and critic model is employed to filter and select "pass" programs (which pass unit tests) and "fail" programs (which do not pass unit tests) from the programs generated by the LM, respectively.

[0027] The "passing" program can then be used to improve program generation: a subsequence from the "passing" program is used as a "seed" that initializes and conditions the LM model to resample new tokens and obtain a new output program, e.g., to generate subsequent tokens after the "seed" to form the output program. In this way, the regenerated program that has been conditioned on the "passing" subsequence can produce a high probability of passing the unit test.

[0028] "Failure" programs can be used to repair program generation. In the "failure" program, a program with a relatively high probability of passing the unit test can be selected (compared with other "failure" programs). These selected program candidates are connected in series with corresponding error information (for example, whether the "failure" program cannot compile, execute or generate correct test results, or whether a specific error has occurred, such as a syntax error, etc.). The program repair module can receive the input connected in series and generate an output code program. In this way, a program regenerated (repaired) is generated based on the information of possible previous errors, and therefore it can have a higher probability of being "repaired" and passing the unit test.

[0029] Figure 1is a simplified block diagram of an exemplary architecture 100 illustrating the use of an actor-critic framework 150 to optimize a pre-trained LM 110 (and a fine-tuned LM 120) for program synthesis according to an embodiment described herein. In one embodiment, the architecture 100 includes a pre-trained / fine-tuned portion 145 of one or more LMs 110 and an actor-critic framework 150. Specifically, the program synthesis task can be formulated as a reinforcement learning (RL) problem, such that the actor-critic RL framework 150 can apply an actor-critic RL method to improve the performance of a pre-trained LM 120 fine-tuned for program synthesis from the pre-trained / fine-tuned stage 145.

[0030] In one embodiment, at stage 145, LM 110 may first be pre-trained on public code data (e.g., from Github) 102. For example, LM 110 may include a transformer model that serves as a skeleton for the program synthesis system described herein. An example of such a pre-trained LM 110 may be a multi-language code-aware language model pre-trained on a large-scale source code corpus curated from Github, such as CodeT5 described in co-pending, commonly owned U.S. non-provisional application Ser. No. 17 / 450.968, filed Aug. 27, 2021, which is hereby expressly incorporated herein by reference in its entirety.

[0031] In one embodiment, public code data 102 may include a Python pre-trained dataset, such as a Github code dataset. Public code data 102 may have compiled public non-personal information from GitHub consisting of licensed Python code (e.g., "mit", "apache-2", "bsd-3-clause", "bsd-2-126clause", "cc0-1.0", "unlicensed", "isc"). The resulting Python dataset (GCPY) has 10.5B tokens and is 10x the CodeSearchNet (CSN) corpus used in the original CodeT5 pre-training.

[0032] In one embodiment, LM 110 can be pre-trained with a pre-training task similar to that used by CodeT5, such as Masked Span Prediction (MSP). Although MSP tasks are beneficial for code understanding, they are very different from program synthesis goals. To mitigate this gap, a pre-training task of Next Token Prediction (NTP) can be used with the pre-trained LM 110. Specifically, a pivot position is uniformly sampled for each code sample, and then the content before the pivot is passed to the encoder of LM 110, and the remaining content is passed to the decoder of LM 110. In order to control the length of the input and output sequences, the pivot can be limited to within 10% to 90% of the original sequence.

[0033] After pre-training, the pre-trained LM 110 can then be fine-tuned for a specific program synthesis task. Following a sequence-to-sequence approach, the pre-trained LM 110 can be fine-tuned using a program synthesis training pair of a natural language problem description 105 (which takes the form of an input sequence D) and a corresponding solution code program 106. In response to the input sequence D, the pre-trained LM 110 can generate an output sequence of a program that can solve the problem. The output at each decoding step t is the distribution over the vocabulary v, via the softmax function is calculated as, where st is the context hidden state at decoding step t.

[0034] Therefore, the model parameters θ of the pre-trained LM 110 can be fine-tuned during training time to maximize the likelihood of the true reference program. Specifically, W = (w1, ..., w T ) is represented as a truth-valued procedure, and the goal is to minimize the cross entropy loss108:

[0035]

[0036] The conditional probability p θ Following the softmax function parameterization above, during inference time, the model can Automatically regressively sample tokens in To generate a sequence of programs.

[0037] In one embodiment, the fine-tuned LM 120 is evaluated against unit tests 112 corresponding to the problem description. Each test includes a pair of inputs and a true value output. In some exemplary real-world program synthesis tasks, exemplary unit tests are typically given as part of the problem specification.

[0038] In one embodiment, the fine-tuned LM 120 is then passed to the actor-critic framework 150 to act as the actor network 130. Specifically, the learned parameters θ of the fine-tuned LM model 120 can be viewed as a stochastic policy that decides actions as predictions for each token in the sampler 133 in response to the input of the problem description 105. After each action, the LM model 120 (or synonymously, the actor network 130) updates its hidden state representations, which the policy uses to determine the next action in the next decoding step. The generated tokens of the sampler 133 can be sent to the critic network 140. At the end of the generated episode (i.e., when the sampler 133 is observed), the sampler 133 is generated. <endoftext>Token), the actor network 130 receives feedback r from the critic network 140 based on the functional correctness of the generated program 133.

[0039] Specifically, for each token w t s Each synthetic sample sequence W sampled by the actor network 130 at decoding time step t s =(w1 s , ..., w T s ), the critic network 140 can determine feedback by checking its functional correctness. On the one hand, the problem description 105 is associated with one or more unit tests 112, and the one or more unit tests contain exemplary test inputs and corresponding outputs that solve the problem description 105. On the other hand, the generated program 133 is also passed to the compiler together with the corresponding unit test 112. Then, the generated program 133 is compiled and executed using the test input from the unit test 112 to generate an execution result. Based on the output of the execution, the synthetic sample sequence W can be used to generate a synthetic sample sequence W. s The feedback r is determined by whether it can be compiled and fully executed and whether the execution result matches the test output in the unit test 112 in the case of successful execution:

[0040]

[0041] The determined reward r can then be used to compute the reinforcement learning training objective, which will minimize the expected reward 135:

[0042]

[0043] To update the actor network 130 (in other words, the fine-tuned LM 120), the policy gradient of the non-differentiable reward r The estimated calculation of is as follows:

[0044]

[0045] The calculated estimated policy gradient may then be used to update the actor network 130 .

[0046] In one embodiment, a "baseline" program can be employed in RL training of the actor network 130. Specifically, a greedy decoding strategy can be used as a baseline, and any generated samples 133 that are better than the baseline are given a positive reward estimate, otherwise a negative reward estimate is given. This relative normalization technique allows the model to explore imperfect programs as long as their rewards are better than the baseline. In other words, given a problem description 105, a baseline model can be used to generate a baseline program sample sequence Wb. The reward r(W) of the baseline is b ) can be compared with r(W s ), and the expected gradient estimate can be calculated to reflect whether the sampled program sequence is better than the baseline program sequence by including the corresponding reward:

[0047]

[0048] At each decoding step t, the greedy decoding baseline is independent of the action w generated by the actor network 130 t s Therefore, the expected gradient term computed with the baseline reward remains the same as the gradient term computed without the baseline reward. However, by introducing the baseline reward term, the high variance of the mini-batch gradient estimates during training can be avoided.

[0049] Figure 2 is a simplified block diagram illustrating an exemplary program synthesis task according to one embodiment described herein. Figure 2 As shown, the exemplary program synthesis task includes a problem specification 105 in natural language, which describes the problem as "printing the maximum length of such substrings that are not palindromes". The corresponding solution program 106 includes a code segment that solves the problem described in the specification 105. The unit test 112 can include an exemplary input-output test pair corresponding to the problem specification 105, for example, when the input = "wuffuw", the output (e.g., the maximum length of a substring that is not a palindrome) = 5.

[0050] The pre-trained language model (LM 120) can be adapted to receive an input sequence as a problem specification 105 in natural language and generate a code sequence as an output program. When the problem specification 105 is passed to a code generator (such as the pre-trained and fine-tuned LM 130), the expected output is a program to be checked against unit tests 112 for functional correctness.

[0051] Figure 3 is a simplified block diagram illustrating an example of a reinforcement learning-based program synthesis framework 300 for an exemplary program synthesis task according to one embodiment described herein. The RL-based program synthesis framework 300 depicts Figure 1 100 . Specifically, in the RL network 300, the fine-tuned LM may act as an actor, which decides actions 216 as predictions for each token in the output program sequence in response to the input of the problem specification 105 according to the learned parameters θ (i.e., the policy) of the fine-tuned LM model. The actions 216 may then be sent to the critic network 140 (which acts as a value function that computes a value 217 (e.g., a policy gradient) based on the current action 216) to update the actor 130. The compiler 204 may act as an environment for the actor 130 and the critic 140, which receives the actions 216 and generates rewards 213, for example, by compiling and executing a sample program sequence consisting of predicted tokens (actions 216) from the actor 130 and comparing the execution results with the unit tests 112. The running state 214 of the environment (compiler 204) may be shared with the actor 130 and the critic 140 to each make its respective predictions.

[0052] Figure 4 is an illustration according to the embodiments described herein Figure 1 4. A simplified block diagram of an exemplary training procedure for the critic network 140 in FIG. The critic model 140 includes a sequence-to-sequence model 402, a linear and softmax operator 404, a max pooling module 406, and a reward estimation module 408.

[0053] In one embodiment, the critic model 140 is parameterized as a neural network with parameters Φ, which is derived from Figure 1 The actor network 130 in receives as the problem description D 105 and the sampling program W s =(w1 s , ..., w T s ) 133. The critic model can be trained as an error predictor that receives the problem specification 105 and the program 133 or 134 as an input sequence, and then predicts one of the four possible test results {compile error; runtime error; test failure; test pass} as described in relation to the reward definition.

[0054] For example, the critic model 140 may include a transformer model, i.e., a sequence-to-sequence model 402, which is smaller in size than the actor model 130 as the base architecture. The context hidden states {h1, ..., h T } is passed to the linear layer 404 and then passed to the max pooling layer 206 for max pooling along the sequence length dimension:

[0055] h pool = Pooling hpool = Pooling(Linear(h1), ..., Linear(h T )). (6)

[0056] The critic's predictions for the unit test results are then calculated as follows:

[0057]

[0058] Thus, the training objective 409 of the critic model 130 parameter Φ can be calculated as the cross entropy loss between the predicted unit test results from the max pooling layer 406 of the critic model 130 and the true unit test results 413:

[0059]

[0060] Here, u represents the sampling program sequence W s 133 is passed to the unit test 112 corresponding to the problem, and then the true value unit test result 413 is given by the compiler. Then, the calculated training target is used to update the critic model 140 (e.g., the max pooling layer 406, the linear and softmax operators 404, and the sequence-to-sequence model 402) via back-propagation.

[0061] After training the critic model 140, the probability distribution to estimate relative to the true unit test output Token-level value of (Note that before the pooling operation, the token-level context ht is used here.) Specifically, the reward estimation module 408 can obtain from the linear and softmax operators 404 And calculate in Represents the probability of a specific unit test outcome from four possible outcomes.

[0062] In some embodiments, in order to improve and stabilize the training process, a baseline program 134 is taken into account, for example, by passing it to the unit test 112 to generate a baseline test result 414. In this way, a relative reward is generated by comparing the sample test result 413 and the baseline test result 414. Specifically, the reward estimation module 408 can then calculate the policy gradient based on the intermediate feedback (using the baseline test result 414 generated by passing the baseline program sequence 134 to the unit test 112):

[0063]

[0064] Note that since the critic model 130 is trained in a supervised learning environment with available truth, the training samples may include perfect (truth) output programs W, such as the solution program 106. These programs may be assigned a default test result u=test passed to train the critic model 140.

[0065] In one embodiment, imitation learning can be employed to first warm-start the pre-trained LM model 110 with only Lce for up to 10 epochs. Then, sample program sequences are obtained from the actor network 130 to train the critic model 140 while keeping the parameters of the actor network 130 frozen. For example, when the actor network is a CodeT5 actor model, the CodeT5-small architecture can be used for the critic model 140, and when the actor model is a GPT variant, the GPT2-small critic architecture can be used for the critic model 140.

[0066] In one embodiment, in addition to the synthetic program 133, the ground truth program 106 of the training samples can also be used to train the critic network 140. These samples are considered perfect programs and always have test passing labels. After training the critic, both Lce and Lrl are applied with equal weights to fine-tune the actor network 130. To optimize the LM actor network 130, a single sample W can be used in each training optimization step. s ~p θ To approximate the expected gradient:

[0067]

[0068] Figure 5 is an illustration of a method for using a Figure 1 Simplified block diagram of the critic sampling (CS) framework 500 for program synthesis using a trained LM (actor network 130). Figure 1 LM130, which is fine-tuned for the task of program synthesis, can be used to generate, improve, and repair programs based on the results of the programs on exemplary unit tests of the corresponding problems. Specifically, a dual strategy (called "critic sampling" (CS)) including a program repair program 560 and a programming improvement program 550 can be implemented to generate and improve programs from both successful cases (program improvement) and failed cases (program repair) during inference. After LM 130 generates output program 533, program improvement program 550 and / or program repair program 560 can each be optionally implemented (e.g., Figure 5 ) or implemented together.

[0069] In one embodiment, a test problem description 505 may be received at the fine-tuned LM 130 during the inference phase. The exemplary unit test input-outputs provided in the input problem description 505 may be used to improve the generation program during inference. For example, exemplary input-output pairs may be extracted from the problem description 505 to form exemplary unit tests 112.

[0070] For each problem description 505, the fine-tuned LM 130 may generate N programs 533. Each generated program 533 may then be passed to an exemplary unit test that is typically embedded as part of the problem specification 505. Specifically, the generated programs 533 may be filtered by the exemplary unit test results at the filtering module 535, such that the filtering module 535 selects the programs that pass the exemplary test as a set 541, and select the remaining failed programs (including programs that cannot be compiled, cannot be executed, or are successfully compiled and executed but cannot generate results that match the exemplary unit test) as a set 542.

[0071] The generated program 533 can be improved by the program 550 to generate the final program 555. Specifically, although by the collection The programs in 541 successfully pass the exemplary test, but there is no guarantee that these programs will successfully perform the final hidden unit test 536. Hidden tests are usually more comprehensive and can include corner cases that test these programs. Therefore, another round of generation can be performed to further improve the programs in the pass set 541.

[0072] In one implementation, from the set Subsequences of these program samples of 541 can be used as prompts (or "seed" sequences) to the actor LM 130. A separate critic model (Φ test ) to guide from the collection 541 of these filtered samples select subsequences. The critic model uses Figure 4 The training objective 409 described above is similar to the training objective described above, but in a binary classification setting with {test failed; test passed} labels. pass ={W1,...,W T } represents a generated sample that passes the exemplary unit test, and the critic model assigns a value to each token of the generated sample:

[0073]

[0074] corresponds to the critic's predicted probability of passing the unit test for subsequences up to t. corresponds to position t with the highest critic assigned value max The sequence at position t max The left subsequence 543 is used as the seed 545 for the next stage. max This seed sequence contains If a token is found, the subsequence is further cut off at the token by removing the token to the right. This is done to pick hints that are likely to produce a successful program in the next round.

[0075] Therefore, subsequence 543 is used as seed 545 to initialize and adjust the (actor) LM 130 to resample until <endoftext>Tokens. In this round, each seed sequence can be stacked N / |P| times for upsampling. This results in the same number of output programs N as in the first round of generation. Finally, the generated N improved programs 555 can be evaluated against the hidden unit test 536.

[0076] In some cases, generating programs that solve problems (especially competition-level programming problems) involves searching a large space of possible programs. Typically, after all programs fail an exemplary test (i.e., ), a complete failure may be observed. Therefore, for these cases, an additional generation step 560 may be employed to first repair the program before improving it.

[0077] In one embodiment, the same critic model (Φ) is used that is also employed in the program improvement process 550. test ), to the module 561 from the failed set 542. Let W fail represent the generated samples from the failure set 542 that failed the exemplary unit test 112, and then the critic model assigns values ​​to the samples:

[0078]

[0079] corresponds to the critic's predicted probability that the program passes the unit test. The top M failed programs with the highest probability are selected 565 and passed to the program repair model w 566.

[0080] In one embodiment, the program repair model 566 is designed as a sequence-to-sequence generation model. The input sequence is the concatenation of the problem description D 505 and the vulnerable program W failure. The additional signal received from the unit test result 112 includes the type of the test result, such as one of a compilation error, a runtime error, a test failure, a test pass, and an error subtype (e.g., a syntax error, an out-of-index error, etc.) may also be included in the input sequence. The error type is extracted from the error trace returned by the compiler.

[0081] To train the program repair model 566, the synthetic samples 133 originally used to train the actor-critic network 150 in RL training are used as the vulnerable program W fail =W s The truth program W 106 can be used as the expected correct program. The training objective of the program repair model is to minimize the cross entropy loss:

[0082]

[0083] Where u is one of {compile error; runtime error; test failed; test passed}, and c is the error subtype. During inference time, each selected failure sequence can be stacked N / M times for upsampling. This results in the same number of output programs N as in the first round of generation. Finally, these N repaired programs generated by the program repair model 561 can be passed to module 543 to apply the code improvement program 550 as described above.

[0084] In one implementation, the program 533 may be generated in small batches to improve efficiency during inference, and kernel sampling with a batch size of N = 200 is used. Note that during program improvement, although resampling using the seed sequence 545 may result in additional computational costs, only a portion of the program needs to be generated during the regeneration phase. In this way, the program improvement phase may not be as expensive as conventional program synthesis.

[0085] Computer environment

[0086] Figure 6 According to some embodiments, Figures 1 to 5 A simplified diagram of a computing device 600 of a reinforcement learning-based program synthesis framework is shown in FIG. Figure 6 As shown, computing device 600 includes a processor 610 coupled to a memory 620. The operation of computing device 600 is controlled by processor 610. Although computing device 600 is shown as having only one processor 610, it should be understood that processor 610 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), etc. in computing device 600. Computing device 600 may be implemented as a standalone subsystem, a board added to a computing device, and / or a virtual machine.

[0087] Memory 620 may be used to store software executed by computing device 600 and / or one or more data structures used during operation of computing device 600. Memory 620 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with a pattern of holes, RAM, PROM, EPROM, flash EPROM, any other memory chip or cartridge, and / or any other media from which a processor or computer is suitable for reading.

[0088] The processor 610 and / or the memory 620 may be arranged in any suitable physical arrangement. In some embodiments, the processor 610 and / or the memory 620 may be implemented on the same board, in the same package (e.g., a system-in-package), on the same chip (e.g., a system-on-chip), etc. In some embodiments, the processor 610 and / or the memory 620 may include distributed, virtualized, and / or containerized computing resources. According to such embodiments, the processor 610 and / or the memory 620 may be located in one or more data centers and / or cloud computing facilities.

[0089] In some examples, memory 620 may include a non-transitory tangible machine-readable medium including executable code that, when executed by one or more processors (e.g., processor 610), may cause one or more processors to perform methods described in more detail herein. For example, as shown, memory 620 includes instructions for a program synthesis module 630 that may be used to implement and / or simulate systems and models, and / or implement any of the methods further described herein. Program synthesis module 630 may receive an input 640 including a natural language problem specification via data interface 615, and generate a code program as output 650.

[0090] In some embodiments, the program synthesis model 630 includes an actor network module 631 (similar to Figure 1 130), critic network module 632 (similar to Figure 1 140 in ) and language model 633 (similar to Figure 1 110 or 120 in ). Can be used with Figures 1 to 5 The details of the program synthesis module 630 and its submodules 631 to 633 and their interactions are discussed in detail.

[0091] In one embodiment, the program synthesis module 630 and its submodules 631 to 633 may be implemented by hardware, software, and / or a combination thereof.

[0092] Some examples of computing devices, such as computing device 400, may include a non-transitory tangible machine-readable medium that includes executable code that, when executed by one or more processors (e.g., processor 410), may cause the one or more processors to perform the processes of the method. Some common forms of machine-readable media that may include the processes of the method are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROMs, EPROMs, flash EPROMs, any other memory chips or cartridges, and / or any other media from which a processor or computer is suitable for reading.

[0093] Figure 7 is suitable for implementing Figures 1 to 5 700 shows a simplified block diagram of a networked system that can be used with a program synthesis framework as described in other embodiments described herein. In one embodiment, block diagram 700 shows a system that includes a user device 710 (which can be operated by a user 740), data provider servers 745, 770, and 780, a server 730, and other forms of devices, servers, and / or software components that operate to perform various methods according to the described embodiments. Exemplary devices and servers can include devices that can be similar to Figure 1 The operating OS described in OS, OS, OS) of computing device 100, independent enterprise-level server, or other suitable device and / or server-based OS. It is understood that Figure 7 The devices and / or servers shown in the drawings may be deployed in other ways, and the operations performed and / or services provided by such devices and / or servers may be combined or separated for a given implementation and may be performed by a greater or lesser number of devices and / or servers. One or more devices and / or servers may be operated and / or maintained by the same or different entities.

[0094] User device 710, data provider servers 745, 770, and 780, and server 730 may communicate with one another via network 760. User 740 (e.g., a driver, system administrator, etc.) may utilize user device 710 to access various features available to user device 710, which may include processes and / or applications associated with server 730 to receive output data anomaly reports.

[0095] The user device 710, the data provider server 745, and the server 730 may each include one or more processors, memories, and other suitable components for executing instructions (such as program code and / or data) stored on one or more computer-readable media to implement the various applications, data, and steps described herein. For example, such instructions may be stored in one or more computer-readable media, such as memory or data storage devices that are internal and / or external to various components of the system 700 and / or accessible via the network 760.

[0096] The user device 710 may be implemented as a communication device that may utilize appropriate hardware and software configured for wired and / or wireless communication with the data provider server 745 and / or the server 730. For example, in one embodiment, the user device 710 may be implemented as an autonomous vehicle, a personal computer (PC), a smart phone, a laptop / tablet computer, a watch with appropriate computer hardware resources, glasses with appropriate computer hardware (e.g., GOOGLE ), other types of wearable computing devices, implantable communication devices, and / or other types of computing devices capable of transmitting and / or receiving data (such as from of ). Although only one communication device is shown, multiple communication devices may function similarly.

[0097] Figure 7 The user device 710 includes a user interface (UI) application 712 and / or other applications 716, which may correspond to executable processes, programs, and / or applications with associated hardware. For example, the user device 710 may receive a message indicating a generated program from the server 730 and display the message via the UI application 712. In other embodiments, the user device 710 may include additional or different modules with required specialized hardware and / or software.

[0098] In various embodiments, the user device 710 includes other applications 716 that may be desired in a particular embodiment to provide features to the user device 710. For example, other applications 716 may include security applications for implementing client-side security features, programming client applications for interfacing with appropriate application programming interfaces (APIs) through the network 760, or other types of applications. Other applications 716 may also include communication applications, such as email, text, voice, social networking, and IM applications, which allow users to send and receive emails, calls, texts, and other notifications through the network 760. For example, other applications 716 may be email or instant messaging applications that receive prediction result messages from the server 730. Other applications 716 may include device interfaces and other display modules that can receive input and / or output information. For example, other applications 716 may include software programs for asset management that can be executed by a processor, including a graphical user interface (GUI) configured to provide an interface to the user 740 to view the generated program.

[0099] The user device 710 may also include a database 718 stored in a transient and / or non-transitory memory of the user device 710, which may store various applications and data and be utilized during the execution of various modules of the user device 710. The database 718 may store user profiles related to the user 740, predictions previously viewed or saved by the user 740, historical data received from the server 730, etc. In some embodiments, the database 718 may be local to the user device 710. However, in other embodiments, the database 718 may be external to the user device 710 and accessible by the user device 710, including a cloud storage system and / or a database accessible via a network 760.

[0100] The user device 710 includes at least one network interface component 719 suitable for communicating with the data provider server 745 and / or the server 730. In various embodiments, the network interface component 719 may include a DSL (e.g., digital subscriber line) modem, a PSTN (public switched telephone network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency, infrared, Bluetooth, and near field communication devices.

[0101] Data supplier server 745 may correspond to a server hosting one or more of databases 703a-n (or collectively referred to as 703) to provide training data sets including public code data to server 730. Database 703 may be implemented by one or more relational databases, distributed databases, cloud databases, etc.

[0102] The data provider server 745 includes at least one network interface component 726 adapted to communicate with the user device 710 and / or the server 730. In various embodiments, the network interface component 726 may include a DSL (e.g., digital subscriber line) modem, a PSTN (public switched telephone network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency, infrared, Bluetooth, and near field communication devices. For example, in one implementation, the data provider server 745 may send asset information from the database 703 to the server 730 via the network interface 726.

[0103] The server 730 may accommodate Figure 1 The program synthesis module 630 and its submodules described in . In some implementations, the module 630 can receive training code data from the database 719 at the data supplier server 745 via the network 760 to generate a program. The generated program can also be sent to the user device 710 via the network 760 for review by the user 740.

[0104] The database 732 may be stored in a transient and / or non-transitory memory of the server 730. In one implementation, the database 732 may store data obtained from the data provider server 745. In one implementation, the database 732 may store parameters of the program synthesis model 630. In one implementation, the database 732 may store previously generated program and problem descriptions, and corresponding input feature vectors.

[0105] In some embodiments, database 732 may be local to server 730. However, in other embodiments, database 732 may be external to server 730 and accessible by server 730, including cloud storage systems and / or databases accessible via network 760.

[0106] The server 730 includes at least one network interface component 733 adapted to communicate with the user device 710 and / or the data provider servers 745, 770, or 780 via the network 760. In various embodiments, the network interface component 733 may include a DSL (e.g., digital subscriber line) modem, a PSTN (public switched telephone network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency (RF), and infrared (IR) communication devices.

[0107] The network 760 may be implemented as a single network or a combination of multiple networks. For example, in various embodiments, the network 760 may include the Internet or one or more intranets, landline networks, wireless networks, and / or other suitable types of networks. Thus, the network 760 may correspond to a small-scale communication network (such as a private network or a local area network) or a larger-scale network (such as a wide area network or the Internet) that can be accessed by various components of the system 700.

[0108] Example Workflow

[0109] Figure 8 is an illustration based on some embodiments described herein Figure 1 800. One or more of the processes of method 800 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors, may cause one or more processors to perform one or more of the processes. In some embodiments, method 800 corresponds to program synthesis module 630 (e.g., Figure 6 to Figure 7 ) operation.

[0110] At step 802, the Figure 6 615 of Figure 7 733) receives a problem specification (e.g., Figure 1 105) and the corresponding solution procedures (e.g. Figure 1 106 in ).

[0111] At step 804, a pre-trained language model (e.g., Figure 1 120) for fine-tuning.

[0112] At step 806, the fine-tuned, pre-trained language model (e.g., Figure 1 130) can be responsive to a problem specification (e.g., Figure 1 105) to generate a sampling program (e.g., Figure 1 133 in ). For example, in one implementation, at a decoding time step t, a language model composed of a fine-tuned, pre-trained language model (e.g., Figure 1 The predicted token w of the sampling procedure governed by the current parameters of 130) t The hidden state representation of the fine-tuned, pre-trained language model may be updated accordingly, and the updated hidden state representation of the fine-tuned, pre-trained language model may be used at the next decoding time step t+1 to generate the next predicted token w for the sampling procedure t+1 .

[0113] At step 808, the critic model (e.g., Figure 1 140) can generate feedback indicating the functional correctness of the sample program based on comparing the execution results of the sample program and the test results of the problem specification. For example, in one implementation, feedback is generated when an end token is generated for the sample program. The sample program (e.g., Figure 1 133 in ) and tests (e.g. Figure 1 112) can be passed to the compiler, and the first feedback value can be determined depending on whether the sample program is successfully compiled and executed and whether the execution result matches the test result of the problem specification, for example, according to equation (2).

[0114] At step 810, the expected policy gradient (e.g., Figure 1 For example, in one implementation, the policy gradient is computed as an estimate of the gradient of the conditional probability of the predicted token based on the reward and the previous predicted token and the problem specification, e.g., according to Equation (4).

[0115] In one implementation, a baseline comparison can be used to compute policy gradients. For example, a baseline program generated by a base model in response to a problem specification can be input to a critic model (e.g., Figure 1 140 in ). The second reward value may be determined depending on whether the baseline program is successfully compiled and executed and whether the execution result matches the test result of the problem specification. In this way, the policy gradient is calculated based on the difference between the first reward value and the second reward value and the gradient of the conditional probability of the prediction token conditioned on the previous prediction token and the problem specification, for example, according to equation (5).

[0116] In another implementation, the policy gradient is computed based on the probability distribution of predicted test outcomes generated by the critic model and the gradient of the conditional probability of the predicted token conditioned on the previous predicted token and the question specification, e.g., according to Eq. (9).

[0117] At step 812, the fine-tuned, pre-trained language model (e.g., Figure 1 130 in the above table).

[0118] In one implementation, a critic model (e.g., Figure 1 140 in ). The critic model can receive a problem specification (e.g., Figure 1 105) of the training sequence and sampling procedure (e.g., Figure 1 133 in ), and generates a predicted test result corresponding to the sampling procedure. The predicted test result is calculated by a softmax operation of the maximum pooled context hidden state of the decoder in the critic model, for example, according to equations (6) to (7). The cross entropy loss is calculated by comparing the predicted test result and the execution result of the sampling procedure, for example, according to equation (8), and the critic model can be updated based on the cross entropy loss.

[0119] Fig. 9 is an illustration based on some embodiments described herein Figure 5 800 corresponds to the program synthesis module 630 (e.g., Figure 6 to Figure 7 ) operation.

[0120] At step 902, the Figure 6 615 of Figure 7 733) in language models pre-trained for program synthesis (e.g., Figure 5 130) receives a problem specification (e.g., Figure 5 505 in the above).

[0121] At step 904, the problem specification (e.g., Figure 2 105) to extract one or more unit test input-output pairs (e.g., Figure 2 112 in the above).

[0122] At step 906, the language model may generate a plurality of program samples (e.g., Figure 5 533 in ).

[0123] At step 908, one or more unit tests (eg, Figure 5 112) is applied to multiple program samples (e.g., Figure 5 533 in ).

[0124] At step 910, a first set of program samples (eg, Figure 5 541 in ) and unsuccessful second set of program samples (e.g., Figure 5 542 in the example). For example, the program samples in the second group include at least one of a compilation error, a runtime error, and failure to pass at least one of the unit tests.

[0125] At step 912, the critic model may determine a value for the second program sample based on a predicted probability that the second program sample in the second group passes one or more unit tests, eg, according to equation (11).

[0126] At step 914, a subset of program samples having the highest values ​​may be selected from the second set (eg, Figure 5 An input sequence is formed by concatenating the problem specification, the selected program sample, and the error information corresponding to the selected program sample (e.g., 566). For example, the error information includes any of the following: a unit test result corresponding to the selected program sample, and an error subtype during compilation or runtime of the selected program sample.

[0127] At step 916, the program repair model may be used to generate a repaired program sample based on the input sequence. For example, the program repair model may be trained by a training target that compares program samples that fail unit tests with a ground-truth program corresponding to the problem specification, the ground-truth program being conditioned on the unit test results and / or error subtypes corresponding to the program sample.

[0128] At step 918, one or more subsequences (e.g., Figure 5 543 in). Each subsequence is a truncated version of the program sample. For example, in one implementation, the critic model can determine the value of each token of the first program sample in the first group based on the predicted probability that the subsequence until the corresponding token passes one or more unit tests, for example, according to equation (10). A specific token with the highest value of the first program sample can be identified, and a subsequence of the first program sample until the specific token can be selected as a subsequence. When the selected subsequence contains a specific token that has a higher probability of failure than the corresponding subsequence passing one or more unit tests, the selected subsequence is further cut off at the specific token.

[0129] At step 920, the language model may generate remaining tokens conditioned on one or more subsequences, e.g., using the subsequences as “seeds” (e.g., Figure 5 545 in the above).

[0130] At step 922, the remaining tokens generated from step 920 may be combined with one or more subsequences to generate one or more improved program samples.

[0131] Example Data Experiment

[0132] exist Figures 1 to 5 The proposed RL-based program synthesis framework and Figures 8 to 9 In the exemplary data experiments of the workflow in , the CodeT5-Large model (770M) is pre-trained from scratch following the T5-Large architecture. Specifically, the code-specific tokenizer is used in the CodeT5 work, and 6 programming languages ​​(PLs) are used in CodeSearchNet (described in "Codesearchnet challenge: Evaluating the state of semantic code search" (abs / 1909.09436) published by Husain et al. in Computing Research Repository (CoRR) in 2019) (CSN) instead of 8 PLs in CodeT5, because the C / C# dataset is not publicly available. Only the pre-training task of masked span prediction (MSP) is applied, so the model does not have to parse the program into an abstract syntax tree (AST) to obtain identifier information.

[0133] In other original pre-trained tasks (such as masked identifier prediction in the original CodeT5 work), the final pre-processing step is required. To further speed up training, data samples are concatenated into batches of size 512 for pre-training with MSP, and the resulting number of tokens is 1.1B. To validate the use of this new, pre-trained CodeT5 as a base model (e.g., Figure 1 130) in the dataset, and evaluate the model on CodeXGLUE.

[0134] The exemplary data experiment was run on kubernetes using 16 A100-40G GPUs on the Google Cloud Platform, and the total pre-training duration was about 21 days. In the first pre-training phase using MSP, a damage rate of 15%, a peak learning rate (LR) of 2e-4, and a batch size of 2048 were used. CSN was pre-trained for 150 epochs (10 days) and then pre-trained on GCPY for 10 epochs (5 days). For the second phase pre-training using NTP, a peak LR of 1e-4 and a batch size of 256 were used, and pre-training was performed for 10 epochs (6 days). For the target, the maximum lengths of the source sequence and the target sequence were set to 768 and 600, respectively. For all experiments, an AdamW optimizer with a weight decay of 0:05 and a linear decay LR scheduler with a warm-up step length of 1000 were used.

[0135] Following (Hendrycks et al., Measuring coding challenge competence with apps, 2021, proceedings of NeurIPS; Chen et al., Evaluating large language models trained on code, 2021, arXiv:2107.03374), models are evaluated using the pass@k metric, which is the percentage of problems solved using k generated programs per problem. Following (Li et al., Competition-level code generation with alphacode, 2022, arXiv:2203.07814), the n@k metric is used, which considers only a subset of n candidates from the k generated programs per problem. The subset of n candidates is typically selected by filtering, i.e., by passing the generated programs through an exemplary test given as part of the problem description.

[0136] Exemplary benchmarks for comparison include the APPS program synthesis benchmark (see Hendrycks et al.) because it has large coding problems of different difficulty levels collected from multiple coding websites. APPS consists of 10,000 coding problems, with a 50-50 training-test split. Each problem is accompanied by an average of 23.2 correct Python programs and 21.2 unit tests. The average length of each problem is 293.2 words, and the average length of each program is 18.0 lines. The data set is classified into three difficulty levels: entry level (3639, training / test=2639 / 1000), interview level (5000, training / test=2000 / 3000) and competition level (1361, training / test=361 / 1000). Each sample includes an average of 20 unit tests to verify the functional correctness of the program. The same preprocessing steps in Hendrycks et al. are used to formulate input sequences from the problem description.

[0137] On APPS, use Figure 1 The pre-trained CodeT5 is fine-tuned using the RL-based framework described in

[11] . To warm-start the CodeT5 model with Lce, a batch size of 64 and warm-up LR from 0 to 2e-5 are used for the first 500 steps and decayed polynomially (power = 0:5) to 1e-5 until the end of 10 epochs, which takes about 30 hours on an A100 GPU. The maximum source and target sequence lengths are set to 600 and 512, respectively.

[0138] Other benchmarks include the MBPP benchmark, a smaller and simpler Python program synthesis dataset for evaluation (described in Austin et al., Program synthesis with large language models, 2021 (arXiv:2108.07732)) (mainly basic programming problems). The dataset contains 974 instances, of which 374 / 90 / 500 instances are used for training / validation / testing respectively, and 10 instances are reserved for few-shot learning. These problems are generally short, usually with only one sentence of natural language description per problem. Each problem is accompanied by 1 correct solution (6.8 lines of code on average) and 3 unit tests in the form of assertion statements to verify functional correctness. Unlike APPS, the unit tests in MBPP are not hidden and are explicitly incorporated into the source sequences of the program synthesis model. This may encourage the model to occasionally overfit these assertion statements via hard-coded if expressions. However, for a fair comparison with the baseline, the source sequences are constructed in the same way as the prior art. Specifically, prepare the input sequence using the same prompt format as in Austin et al.: problem description + "Your code should satisfy these tests:" + 3 assert statements.

[0139] Experiments were conducted on MBPP in both zero-shot (Section 4.5) and full fine-tuning settings. For fine-tuning CodeT5, due to the smaller training set of MBPP, the model was fine-tuned for 60 epochs with a constant LR of 2e-5 and a batch size of 32, which takes less than 30 minutes on an A100. The maximum source and target lengths were set to 382 and 306, respectively.

[0140] Exemplary baselines include GPT2 (Radford et al., "Language models are unsupervised multitask learners" published in "OpenAI blog" Vol. 1, Issue 8, Page 9, 2019), GPT-Neo (Black et al., "GPT-NEO: Large scale autoregressive language modeling with mesh-tensorflow" published in 2021, URL is https: / / doi.org / 10.5281 / zenodo.5297715), and GPT3 (Brown et al., "Language models are few-shot learners" published in "Advances in neural information processing systems" Vol. 33, Pages 1877-1901, 2020) to compare with the RL-based framework described in this article (called "CodeRL"). The results are also compared with Codex (see Chen et al.) and AlphaCode (see Li et al.). Note that by default, the results for pre-trained LMs (except Codex and GPT3) are from models fine-tuned on APPS using only the standard loss Lce. Since CodeRL is model-agnostic, it can also be integrated with GPT variants such as GPT-J and GPT-Neo.

[0141] Fig.10 (a) shows that CodeRL with the CodeT5 model can achieve significant performance gains, outperforming many pre-trained LMs of much larger sizes. Specifically, CodeRL achieves new SOTA results: 2:69% passing @1, 6:81% passing @5, and 20:98% passing @1000. Fig.10 (b) shows that when evaluated on a subset of filtered code samples, CodeRL+CodeT5 can achieve SOTA results: 8.48% for 1@k and 12.62% for 5@k.

[0142] Second, the benefit of up-sampled generation is observed when increasing the number of generated samples k from 1 to 1000. Note that while CodeRL incurs additional computational cost during inference with CS, CodeRL only requires a much lower k to achieve comparable performance with other models.

[0143] Specifically, CodeRL performs as well as AlphaCode only when k=1000, but AlphaCode has a much larger generation budget of k=50000. Fig.10 (b) also shows that fine-tuning can significantly improve model performance on challenging programming tasks at interview and competition levels. Specifically, Codex, which is not fine-tuned on APPS and tested in a few-shot setting, can achieve good n@1000 results, but the model performs poorly on synthetic tasks at interview and competition levels. This observation suggests a significant gap between the pre-training stage and downstream synthetic tasks.

[0144] Fig.11 Results for estimating rewards for code samples trained with CodeT5-770M by different methods are shown. Overall, the CodeRL objective with relative token-level reward estimation via a critic model (Model D) achieves the best performance on Pass@1 and Pass@5. Second, we note that using absolute rewards without a baseline (Model B) can lead to the largest performance drop, as this approach severely penalizes all incorrect samples (even though they may still be better than the naive baseline). Therefore, considering relative reward estimates that can effectively exploit imperfect code can lead to better synthetic systems. Third, simply assigning the same reward to all tokens in a code sample without a critic model (Model A) is disadvantageous because these reward estimates are too constrained to be used as feedback signals for RL training. For example, a program is considered incorrect simply because of an additional space character (which can cause indentation errors in Python programs). Simply assigning the same reward to all tokens in a program will severely penalize the correct parts of the program sequence. Finally, we conducted experiments with a distance-based critic that assumes that token values Decays linearly from t=1 to t=T (Model C). The lower performance suggests that there is benefit in training a critic network to compute the feedback rather than relying on rule-based methods.

[0145] Fig.12 Results for different combinations of Lce and Lrl are shown. Since CodeRL is model-agnostic, experiments were performed on both CodeT5 and GPT-Neo. Note that in these experiments, Lce and Lrl were applied to models that had been warm-started / fine-tuned with Lce for up to 10 epochs. First, when only Lrl was used, the problem of vanishing gradients during fine-tuning was observed. As a result, the final model actually deteriorated and resulted in reduced performance. Second, by performing further fine-tuning using only Lce, although the loss improved during training time, the model performance did deteriorate during test time. Therefore, these models are expected to overfit the training data, as shown in Fig.16 This is similarly observed in our analysis of the pre-trained models.

[0146] Additionally, we observe that the naive approach of Lce with synthetic samples Ws (all samples are treated as correct codes with r(Ws)=1) still leads to some performance improvement of GPT-Neo on Pass@5. However, in all other cases, the training strategy does not work as well as considering the critic model to estimate its reward by the test results of Ws. Finally, we observe that using both Lce and Lrl achieves more consistent performance improvements on Pass@1 and Pass@5 for GPT-Neo and CodeT5 models overall.

[0147] Fig.13 Ablation results of applying Critic Sampling (CS) on the CodeT5 model during inference are shown. Different combinations of program improvement and repair steps are tested. Overall, a positive impact of CS on all metrics is observed in the case of combining program improvement and repair, with particularly more significant gains on passing@1000. Note that program improvement alone can help bring performance gains, but its impact on the 1@1000 metrics is reduced. Note that n@k measures the solution rate in a subset P filtered from k samples. Since program improvement will technically increase the size of the subset, the n@k metric will consider a much larger number of options for n samples than before. This will normalize n@k by a larger pool of n candidate sets, resulting in less impact of program improvement on model performance.

[0148] Second, when integrating program improvement with program repair (for problems with P=;), performance in all metrics is further improved. Interestingly, when experimenting with different top-M selection schemes, we find that the overall performance is best when M=1, and the performance starts to degrade from M=2 to M=4 (except for the pass@200 results). This observation suggests that there are benefits to using a critic model to focus on the best candidates for program repair rather than selecting multiple program candidates. In addition, as M increases, each program candidate will have a smaller batch size (i.e., N=M). This results in a lower chance that the program repair model correctly repairs and generates a correct program.

[0149] In one embodiment, the data experiments study a subset of the APPS test split that contains test samples at the highest difficulty level (ie, competitive programming tasks). Fig.14 Results are shown for @k and n@k, where k ranges from 1 to 200 and n = f1;5g, for CodeRL + CodeT5 and CodeT5 alone. Since CodeRL is model-agnostic, we also integrate it with GPT-J and report the results. To focus on the impact of RL optimization, the model is compared to using only kernel sampling without the CS procedure during test time.

[0150] Fig.14 We show that the performance gains on GPT-J and CodeT5 are fairly consistent. Specifically, as k increases, CodeRL's performance gains on both GPT-J and CodeT5 models become more significant. We attribute these gains to the CodeRL learning objective Lrl, which encourages the model to explore code solutions drawn from the model's sampling distribution. During test time, as the k sampling budget increases, the model is allowed to generate different code solutions, and the impact of Lrl becomes more significant.

[0151] In one embodiment, the performance of the synthesis system is related to the quality of the underlying model. Fig.15 Results for CodeT5 are reported for different configurations of model size, pre-training data, and pre-training targets. For fair comparison, all models are fine-tuned / warm-started for up to 12 epochs on APPS using only Lce. It is observed that scaling up the number of model parameters (from 60M to 770M) significantly improves model performance on downstream synthesis tasks. When the pre-training data is improved by adding the GCPY dataset (10x of the CSN dataset), good performance improvements are observed, from 1:3 to 1:56 via @1 and from 1:72 to 2:06 via @5. Finally, by combining pre-training targets from masked span prediction (MSP) and next token prediction (NTP), the model is better able to adapt to downstream synthesis.

[0152] Fig.16 The performance of CodeT5 model variants by difficulty level of programming tasks under fine-tuning epochs is shown. Note that in these experiments, data experiments were compared between CodeT5 model variants only by pre-training strategy, so Lce was only engaged in the fine-tuning stage on APPS. Consistent with our previous analysis, both enhanced pre-training data (more data with GCPY) and pre-training targets (with NTP targets) generally improved model performance throughout the training epoch. In addition, as noted by the analysis of learning targets, using only Lce often leads to overfitting performance, typically after 10 epochs in our case. Therefore, in order to further fine-tune the large-scale LM, it is beneficial to adopt the RL target Lrl to exploit synthetic training samples and avoid overfitting the model.

[0153] Fig.17 We report the results of our CodeRL+CodeT5 on the MBPP benchmark compared to fine-tuned GPT models up to 137B in size. CodeRL+CodeT5 (ZS) is trained on APPS and then evaluated on MBPP in the zero-shot setting. We observe that CodeRL with CodeT5, which has a much smaller model size, produces surprisingly good zero-shot performance, setting a new SOTA result of 63.0% passing@80 relative to 61.4% passing@80 for GPT-137B. This validates CodeRL's strong zero-shot transfer capability for unknown tasks.

[0154] A common concern about transfer learning is that the source (APPS) and target (MBPP) tasks may overlap in their training data, which may cause the source model to tend to remember these substantially similar data when applied to the target task. To address this concern, we followed Austin et al. to analyze how many lines of code appeared in the APPS training set and the MBPP programs. For the analysis, code comments were discarded, and whitespace was normalized for each line, and then lines that appeared more than twice anywhere in MBPP were excluded, as these lines may be common Python keywords such as return and break.

[0155] Fig.18 The absolute number of duplicate lines in the MBPP programs (left) and the relative fraction of duplicate lines (right) are shown. As can be seen, the overlap between APPS and MBPP appears to be minimal. Only 12.6% of the MBPP programs have more than half of their lines matching somewhere in the APPS training data. Furthermore, more than half (514 out of 974) of the programs have zero overlap, and 90.9% of the programs have no more than 3 lines overlapping the APPS training set. If the lines are further required to be continuous, then there are no more than 2 continuous duplicate lines.

[0156] Fig.19 The average percentage of generated programs for each problem is shown, grouped by their test results. Specifically, CodeT5 or CodeRL+CodeT5 is used to generate programs, and 200 generated programs are randomly selected for each test sample in the APPS test split. The programs are passed to the exemplary unit test or the hidden unit test, and the output programs are grouped by their test results. The results are classified according to the definition in equation (2), including compilation errors, runtime errors, test failures, and test passes.

[0157] First, on both the exemplary unit tests and the hidden unit tests, it is observed that integrating CodeRL can increase the probability that the program can pass the tests and reduce the probability that it fails one or more unit tests. In the entry-level programming problems, the probability of passing the unit tests is improved more significantly.

[0158] Second, notice that the percentage of CodeRL-generated programs with compilation errors has decreased, with a greater impact on interview- and competition-level problems. Although CodeRL programs are less likely to have compilation errors, these programs still suffer from runtime errors. This results in a higher probability that CodeRL programs contain runtime errors.

[0159] Note that in the example unit test ( Fig.19 (a)) and hidden unit tests ( Fig.19 (b)). This observation suggests that the exemplary tests are not as comprehensive as the hidden tests and thus limit the positive impact of the CodeRL generators due to false positives.

[0160] Fig. 20 Examples of programming problems from the APPS benchmark and the corresponding programs generated by the CodeT5 variants are shown. Specifically, a CodeT5 model fine-tuned only by Lce is compared with another model following the CodeRL framework, based on CodeT5 pre-trained on the same basis (pre-trained with GCPY data and NTP targets). In CodeRL+CodeT5, the program before and after applying the CS program is shown. It is observed that applying CodeRL can generate more appropriate programs, and its functional correctness is further improved using the CS program. For example, in Fig. 20 In the example above, the CodeT5 model misunderstood the problem and only focused on finding the greatest common divisor between a and b. In contrast, the CodeRL model avoided this mistake and solved the problem of finding the greatest common divisor between the factorials of a and b.

[0161] CodeRL was also found to improve the complexity of generated programs, an important quality in complex programming problems. Fig. 20 In the interview-level program in , without applying CS, the generated program is functionally correct, but fails during execution due to a timeout error. The program simply computes separate factorials of both a and b, which will slow down execution in scenarios where a or b is extremely large. Applying CS can adjust the model on a portion of the previous program and (re)generate new tokens to produce a more efficient program. Fig. 20 In the example, the factorial calculation is performed on the minimum value (a, b) to improve the efficiency of the program. Therefore, the final program can pass all hidden unit tests (including those with extremely large values) without timeout errors.

[0162] This specification and the drawings showing aspects of the invention, embodiments, implementations or applications should not be considered restrictive. Various mechanical, compositional, structural, electrical and operational changes may be made without departing from the spirit and scope of this specification and claims. In some cases, in order not to obscure the embodiments of the disclosure, known circuits, structures or technologies are not shown or described in detail. Similar reference numerals in two or more figures represent identical or similar elements.

[0163] In this specification, the specific details of some embodiments consistent with the disclosure are described. Many specific details are described to provide a thorough understanding of the embodiments. However, it is obvious to those skilled in the art that some embodiments can be put into practice without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative rather than restrictive. Those skilled in the art will appreciate that other elements fall within the scope and spirit of the disclosure, although other elements are not specifically described herein. In addition, in order to avoid unnecessary repetition, unless specifically described otherwise or when one or more features will make the embodiment inoperative, one or more features shown and described in association with an embodiment may be incorporated into other embodiments.

[0164] Although exemplary embodiments have been shown and described, various modifications, changes and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without corresponding use of other features. Those of ordinary skill in the art will recognize many variations, substitutions and modifications. Therefore, the scope of the present invention should be limited only by the appended claims, and the claims should be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.< / endoftext> < / endoftext>

Claims

1. A method for a reinforcement learning framework for program synthesis, the method comprising: receiving a problem specification and a corresponding solution program via an input interface; Fine-tuning a pre-trained language model based on the problem specification and the corresponding solution procedure; generating, via the fine-tuned, pre-trained language model, a sampling program in response to the problem specification at a decoding time step; generating, by a critic model, feedback indicating functional correctness of the sample program based on comparing execution results of the sample program with test results of the problem specification; computing a desired policy gradient of the feedback taking into account current parameters of the fine-tuned, pre-trained language model; and The fine-tuned, pre-trained language model is updated according to the policy gradient.

2. The method according to claim 1, wherein the sampling program is generated by: generating, at the decoding time step, a prediction token for the sampling procedure governed by the current parameters of the fine-tuned, pre-trained language model; updating a hidden state representation of the fine-tuned, pre-trained language model; and At the next decoding time step, the next predicted token of the sampling procedure is generated using the updated hidden state representation of the fine-tuned, pre-trained language model. The method of claim 2 , wherein the feedback is generated when an end token is generated for the sample program.

4. The method of claim 1, wherein the feedback is generated by: passing the sample program and the test to a compiler; and A first feedback value is determined depending on whether the sample program is successfully compiled and executed and whether the execution result matches the test result of the problem specification.

5. The method of claim 4, wherein the policy gradient is computed as an estimate of a gradient based on the reward and a conditional probability of a predicted token conditioned on previous predicted tokens and the problem specification.

6. The method according to claim 4, further comprising: inputting into the critic model a baseline program generated by a base model in response to the problem specification; and A second feedback value is determined depending on whether the baseline program is successfully compiled and executed and whether the execution result matches the test result of the problem specification.

7. The method of claim 6, wherein the policy gradient is calculated based on a difference between the first reward value and the second reward value and a gradient of a conditional probability of a prediction token conditioned on previous prediction tokens and the problem specification.

8. The method of claim 1, wherein the critic model is trained by: receiving, by the critic model, a training sequence of the problem specification and the sampling procedure; generating, by the critic model, predicted test results corresponding to the sampling procedure; Calculating a cross entropy loss by comparing the predicted test result with the execution result of the sampling program; and Based on the cross entropy loss, the critic model is updated.

9. The method of claim 8, wherein the predicted test result is calculated by a softmax operation of the maximum pooled context hidden state of a decoder in the critic model.

10. The method of claim 9, wherein the policy gradient is calculated based on the probability distribution of the predicted test outcome generated by the critic model and the gradient of the conditional probability of the predicted token conditioned on previous predicted tokens and the problem specification.

11. A system for a reinforcement learning framework for program synthesis, the system comprising: an input interface that receives a problem specification and a corresponding solution program; a memory storing a plurality of processor executable instructions; A processor that reads and executes the plurality of processor-executable instructions to perform operations, the operations comprising: Fine-tuning a pre-trained language model based on the problem specification and the corresponding solution procedure; generating, via the fine-tuned, pre-trained language model, a sampling program in response to the problem specification at a decoding time step; generating, by a critic model, feedback indicating functional correctness of the sample program based on comparing execution results of the sample program with test results of the problem specification; computing a desired policy gradient of the feedback taking into account current parameters of the fine-tuned, pre-trained language model; and The fine-tuned, pre-trained language model is updated according to the policy gradient.

12. The system of claim 11, wherein the sampling program is generated by: generating, at the decoding time step, a prediction token for the sampling procedure governed by the current parameters of the fine-tuned, pre-trained language model; updating a hidden state representation of the fine-tuned, pre-trained language model; and At a next decoding time step, a next predicted token for the sampling procedure is generated using the updated hidden state representation of the fine-tuned, pre-trained language model.

13. The system of claim 12, wherein the feedback is generated when an end token is generated for the sample program.

14. The system of claim 11, wherein the feedback is generated by: passing the sample program and the test to a compiler; and A first feedback value is determined depending on whether the sample program is successfully compiled and executed and whether the execution result matches the test result of the problem specification.

15. The system of claim 14, wherein the policy gradient is computed as an estimate of a gradient based on the reward and a conditional probability of a predicted token conditioned on previous predicted tokens and the problem specification.

16. The system of claim 14, wherein the operations further comprise: inputting into the critic model a baseline program generated by a base model in response to the problem specification; and A second feedback value is determined depending on whether the baseline program is successfully compiled and executed and whether the execution result matches the test result of the problem specification.

17. The system of claim 16, wherein the policy gradient is calculated based on a difference between the first reward value and the second reward value and a gradient of a conditional probability of a prediction token conditioned on previous prediction tokens and the problem specification.

18. The system of claim 11, wherein the critic model is trained by: receiving, by the critic model, a training sequence of the problem specification and the sampling procedure; generating, by the critic model, predicted test results corresponding to the sampling procedure; Calculating a cross entropy loss by comparing the predicted test result with the execution result of the sampling program; and Based on the cross entropy loss, the critic model is updated.

19. The system of claim 18, wherein the predicted test result is calculated by a softmax operation of a max-pooled context hidden state of a decoder in the critic model, and wherein the policy gradient is calculated based on the probability distribution of the predicted test outcome generated by the critic model and the gradient of the conditional probability of the predicted token conditioned on the previous predicted token and the problem specification.

20. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for a reinforcement learning framework for program synthesis, the processor-executable instructions being executed by one or more processors to perform operations comprising: receiving a problem specification and a corresponding solution program via an input interface; Fine-tuning a pre-trained language model based on the problem specification and the corresponding solution procedure; generating, via the fine-tuned, pre-trained language model, a sampling program in response to the problem specification at a decoding time step; generating, by a critic model, feedback indicating functional correctness of the sample program based on comparing execution results of the sample program with test results of the problem specification; computing a desired policy gradient of the feedback taking into account current parameters of the fine-tuned, pre-trained language model; and The fine-tuned, pre-trained language model is updated according to the policy gradient.

21. A method for program synthesis, the method comprising: Receiving, via an input interface, a question specification at a language model pre-trained for program synthesis; extracting one or more unit test input-output pairs from the problem specification; generating, by the language model, a plurality of program samples from the problem specification; applying one or more unit tests to the plurality of program samples based on the one or more unit test input-output pairs; determining, from the plurality of program samples, a first group of program samples that pass the one or more unit tests; and One or more improved program samples are regenerated by the language model based on information from the first set of program samples.

22. The method of claim 21, wherein regenerating one or more improved program samples from the language model comprises: selecting one or more subsequences from the first set of program samples via critic scores, each subsequence being a truncated version of a program sample; and Subsequent tokens conditioned on the one or more subsequences are regenerated by the language model to form the one or more improved program samples.

23. The method of claim 22, wherein the one or more subsequences are selected by: determining, by a critic model, a value for each token of a first program sample in the first set based on a predicted probability until a subsequence of the corresponding token passes the one or more unit tests; determining a particular token of said first program sample having a highest value; and A subsequence of the first program sample up to the specific token is selected as a subsequence.

24. The method according to claim 23, further comprising: determining that the selected subsequence contains a particular token, up to the particular token, the corresponding subsequence having a higher probability of failing than passing the one or more unit tests; and The selected subsequence is cut off at the specific token.

25. The method of claim 21, further comprising: A second group of unsuccessful program samples is determined from the plurality of program samples, the second group of program samples comprising at least one of a compilation error, a runtime error, and a failure to pass at least one of the unit tests.

26. The method according to claim 25, further comprising: determining, by the critic model, a value of a second program sample in the second group based on a predicted probability that the second program sample passes the one or more unit tests; selecting a subset of program samples having the highest values ​​from said second set; forming an input sequence by concatenating the problem specification, the selected program sample, and the error information corresponding to the selected program sample; A repaired program sample is generated by a program repair model based on the input sequence.

27. The method according to claim 26, wherein the error information comprises any one of the following: unit test results corresponding to the selected program sample; and The subtype of errors during compilation or runtime of the selected program sample.

28. The method of claim 26, wherein the program repair model is a sequence-to-sequence model.

29. The method of claim 26, wherein the program repair model is trained by a training objective, wherein the training objective compares program samples that fail the unit test with a true program corresponding to the problem specification, wherein the true program is conditioned on a unit test result and / or an error subtype corresponding to the program sample.

30. The method of claim 25, further comprising: generating, by a program repair model, one or more repaired program samples using the error information from the second set of program samples; and The one or more repaired program samples are improved by the language model.

31. A system for program synthesis, the system comprising: an input interface that receives a question specification at a language model pre-trained for program synthesis; a memory storing a plurality of processor executable instructions; A processor that reads and executes the plurality of processor-executable instructions to perform operations, the operations comprising: extracting one or more unit test input-output pairs from the problem specification; generating, by the language model, a plurality of program samples from the problem specification; applying one or more unit tests to the plurality of program samples based on the one or more unit test input-output pairs; determining a first set of program samples from the plurality of program samples that pass the one or more unit tests; and One or more improved program samples are regenerated by the language model based on information from the first set of program samples.

32. The system of claim 31, wherein the operation of regenerating one or more improved program samples from the language model comprises: selecting one or more subsequences from the first set of program samples via critic scores, each subsequence being a truncated version of a program sample; and Subsequent tokens conditioned on the one or more subsequences are regenerated by the language model to form the one or more improved program samples.

33. The system of claim 32, wherein the one or more subsequences are selected by: determining, by a critic model, a value for each token of a first program sample in the first set based on a predicted probability until a subsequence of the corresponding token passes the one or more unit tests; determining a particular token of said first program sample having a highest value; and A subsequence of the first program sample up to the specific token is selected as a subsequence.

34. The system of claim 33, wherein the operations further comprise: determining that the selected subsequence contains a particular token, up to the particular token, the corresponding subsequence having a higher probability of failing than passing the one or more unit tests; and The selected subsequence is cut off at the specific token.

35. The system of claim 31 , wherein the operations further comprise: A second group of unsuccessful program samples is determined from the plurality of program samples, the second group of program samples comprising at least one of a compilation error, a runtime error, and a failure to pass at least one of the unit tests.

36. The system of claim 35, wherein the operations further comprise: determining, by the critic model, a value of a second program sample in the second group based on a predicted probability that the second program sample passes the one or more unit tests; selecting a subset of program samples having the highest values ​​from said second set; forming an input sequence by concatenating the problem specification, the selected program sample, and error information corresponding to the selected program sample; A repaired program sample is generated by a program repair model based on the input sequence.

37. The system according to claim 36, wherein the error information comprises any of the following: unit test results corresponding to the selected program sample; and The subtype of errors during compilation or runtime of the selected program sample.

38. The system of claim 36, wherein the program repair model is trained by a training objective that compares program samples that fail the unit test with a true program corresponding to the problem specification, the true program being conditioned on a unit test result and / or an error subtype corresponding to the program sample.

39. The system of claim 35, wherein the operations further comprise: generating, by a program repair model, one or more repaired program samples using the error information from the second set of program samples; and The one or more repaired program samples are improved by the language model.

40. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for program synthesis, the processor-executable instructions being executed by one or more processors to perform operations comprising: Receiving, via an input interface, a question specification at a language model pre-trained for program synthesis; extracting one or more unit test input-output pairs from the problem specification; generating, by the language model, a plurality of program samples from the problem specification; applying one or more unit tests to the plurality of program samples based on the one or more unit test input-output pairs; determining, from the plurality of program samples, a first group of program samples that pass the one or more unit tests; and One or more improved program samples are regenerated by the language model based on information from the first set of program samples.

Images