Generating Computer Code from Task Descriptions Using Neural Networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating computer code using neural networks. One of the methods includes receiving descriptive data describing a computer programming task, receiving a first set of inputs for the computer programming task, generating a plurality of candidate computer programs by sampling a plurality of output sequences from a set of one or more generative neural networks, for each candidate computer program in the subset of candidate computer programs and for each input in the first set, executing the candidate computer program on the input to generate an output, and selecting one or more computer programs from the candidate computer programs as a composed computer program for performing the computer programming task based at least in part on the output generated by executing the candidate computer programs in the subset on the inputs in the first set of inputs.

Description

[Technical field]

[0001] This specification relates to using neural networks to generate computer code. [Background technology]

[0002] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to the next layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer", arXiv preprint arXiv:1910.10683, 2019 [Non-Patent Document 2] Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, “Towards a human-like open-domain chatbot,” CoRR, abs / 2001.09977, 2020 [Non-Patent Document 3] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, "Attention Is All You Need", arXiv preprint arXiv: 1706.03762, 2017 [Non-Patent Document 4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018. Summary of the Invention [Means for solving the problem]

[0004] This specification describes a system implemented as a computer program on one or more computers at one or more locations that uses neural networks to generate computer program code. Specifically, the system receives input including descriptive data describing computer programming tasks, and generates as output one or more computer programs that, when executed, perform the computer programming tasks described in the descriptive data.

[0005] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages.

[0006] Generating code that solves a given task requires searching within a large structured space of possible computer programs with an extremely sparse reward signal. For example, a single character edit can completely change program behavior even if it does not cause a crash, and solutions can look dramatically different for the same problem, making it a difficult challenge to determine whether a partial or incorrect program is useful. Thus, generating an entire program in a general-purpose programming language such as C++ or Python starting from a natural language task description remains an open problem.

[0007] This specification describes techniques for efficiently leveraging a set of one or more generative neural networks to accurately generate computer programs conditioned on natural language descriptions. Specifically, the described techniques address the problem of searching a large structured space by using generative neural networks, e.g., Transformer models, to generate a large set of program samples, and then using the corresponding inputs to filter, cluster, or both the results to obtain a small set of candidates that will be proposed as synthesized computer programs. In some cases, the techniques utilize "metadata conditioning" to improve the diversity of the generated samples.

[0008] Moreover, the described system can pre-train a generative neural network on code sequences and then fine-tune it on a set of task-specific data to ensure that the generative neural network can generate high-quality candidates even when the amount of task-specific training data is limited.

[0009] Additionally, the described techniques are designed to be particularly efficient when implemented on parallel processing hardware. Specifically, the system can perform the generation of candidates and the sampling of output sequences in parallel. For example, the system can perform the generation and sampling in parallel using multiple parallel processing devices, e.g., CPUs, GPUs, TPUs, or other ASICs, FPGAs, etc. As a specific example, the system can deploy multiple instances of each of one or more generative neural networks across those devices. For example, the system can deploy each instance on a different device, or use model parallelism techniques to deploy the instances across multiple devices. For each instance, the system can then sample candidates as described above, using the instance in parallel with each other instance. The system can further parallelize the code generation process by evaluating the candidates in parallel on the same device on which they were generated.

[0010] One example method described herein is executed by one or more computers and includes the steps of receiving descriptive data describing a computer programming task, the descriptive data comprising a plurality of text tokens; receiving a first set of inputs for the computer programming task; generating a plurality of candidate computer programs by sampling a plurality of output sequences from a set of one or more generative neural networks, each generative neural network in the set configured to receive an input sequence including the plurality of text tokens in the descriptive data and process the input sequence to generate an output sequence comprising a plurality of computer code tokens that specify a candidate computer program for performing the computer programming task described by the descriptive data; for each candidate computer program in the subset of candidate computer programs and for each input in the first set, executing the candidate computer program on the input to generate an output; and selecting one or more computer programs from the candidate computer programs as a composed computer program for performing the computer programming task based at least in part on the output generated by executing the candidate computer programs in the subset on the inputs in the first set of inputs.

[0011] The method may include receiving a new input for a computer programming task, and executing one or more of the composed computer programs against the new input to generate an output for the computer programming task for the new input. The method may further include receiving, for each input in the first set, a respective expected output generated by executing the computer programming task against the input. The selecting step may include determining, for each candidate computer program in the subset and for each input in the first set, whether an output generated by executing the candidate computer program against the input matches a respective expected output for the input, selecting, for each input in the first set, as an initial set of computer programs only those candidate computer programs that, when executed against the input, generated an output that matched the respective expected output for the input, and selecting one or more respective computer programs from the initial set of computer programs as the composed computer program. The subset of candidate computer programs may include all of the plurality of candidate computer programs.

[0012] The selecting step may include clustering the candidate computer programs in the subset into a plurality of clusters based on outputs generated by executing the candidate computer programs on the inputs in the first set, and selecting one or more respective candidate computer programs from each of one or more of the clusters as the composite computer program. The selecting one or more respective computer programs from each of one or more of the clusters as at least one of the composite computer programs may include selecting one or more respective computer programs from each of a predetermined number of largest clusters of the plurality of clusters as at least one of the composite computer programs. The receiving a first set of inputs for the computer programming task may include processing one or more description sequences, each including a plurality of text tokens from the description data, using a test input generation neural network configured to process each description sequence to generate as output one or more test inputs for the computer programming task described by the description sequence. In one example, the subset of candidate computer programs may include only those candidate computer programs that were not filtered out from the multiple candidate computer programs based on whether, for each candidate computer program, and for each input in the second set of inputs, outputs generated by executing the candidate computer program on the input matched respective expected outputs for the input.

[0013] The set of one or more generative neural networks may include only a single generative neural network. Alternatively, the set of one or more generative neural networks may include multiple generative neural networks. Each generative neural network in the set may have a combination of values ​​for a set of one or more properties that differs from each other neural network in the set. The properties may include one or more of: (i) training data used to train the generative neural network, (ii) a number of parameters of the generative neural network, or (iii) a sampling temperature for sampling an output sequence from the generative neural network. One or more of the generative neural networks in the set may include an encoder neural network configured to process an input sequence to generate an encoded representation of the input sequence, and a decoder neural network configured to process the encoded representation to generate an output sequence. The encoder neural network in the set may be a transformer encoder that applies self-attention across the input sequence. The decoder neural network in the set may be an autoregressive transformer decoder that applies cross-attention to the encoded representation.

[0014] Generating a plurality of candidate computer programs by sampling a plurality of output sequences from a set of one or more generative neural networks may include repeatedly performing operations including generating a current input sequence from descriptive data and sampling one or more output sequences by processing the current input sequence using one of the generative neural networks in the set. The operations may be performed in parallel across multiple hardware devices. Each input sequence may comprise a plurality of text tokens from the descriptive data and one or more tokens identifying a programming language. Generating a current input sequence from the descriptive data may include sampling a programming language from a distribution across the set of programming languages ​​and including one or more tokens identifying a programming language in the current input sequence.

[0015] Each input sequence may comprise a plurality of text tokens from the descriptive data and one or more tokens indicating whether an output sequence generated by the generative neural network should specify a correct answer or an incorrect answer for the computer programming task. Generating the current input sequence from the descriptive data may include including one or more tokens indicating that an output sequence generated by the generative neural network should specify a correct answer for the computer programming task.

[0016] Each input sequence may comprise a plurality of text tokens from the descriptive data and zero or more tags, each comprising one or more tokens, that identify an algorithm to be implemented by the output sequence generated by the generative neural network. Generating a current input sequence from the descriptive data may include sampling one or more tags from a distribution over the set of tags and including the sampled tags in the current input sequence.

[0017] Each input sequence may comprise a plurality of text tokens from the descriptive data and one or more tokens that specify a difficulty rating for the computer programming task. Generating a current input sequence from the descriptive data may include selecting a difficulty rating for the computer programming task and including, in the current input sequence, one or more tokens that specify the selected difficulty rating. Selecting a difficulty rating for the computer programming task may include selecting a fixed difficulty rating that is the same for all computer programming tasks after training of the generative neural network. Selecting a difficulty rating for the computer programming task may include sampling a difficulty rating from a distribution over the set of difficulty ratings.

[0018] Each generative neural network may be pre-trained for language modeling purposes based on a first set of training data and fine-tuned for code generation purposes based on a second set of training data including a plurality of training examples, each of which includes (i) a training input sequence representing a computer programming task and (ii) a training output sequence representing a computer program. The first set of training data may comprise a set of sequences each representing a segment of computer code. The plurality of training examples may include a first subset of training examples in which the training output sequence in the training examples is a correct answer to the computer programming task represented by the training input sequence in the training examples, and a second set of training examples in which the training output sequence in the training examples is not a correct answer to the computer programming task represented by the training input sequence in the training examples. The code generation objective may be GOLD with tempering.

[0019] The text tokens and the computer code tokens may be selected from the same vocabulary of tokens.

[0020] Example systems described herein include one or more computers and one or more storage devices that store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of any of the methods described herein.

[0021] In another example described herein, one or more computer storage media may store instructions that, when executed by one or more computers, cause the one or more computers to perform operations of any of the methods described herein.

[0022] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram of an example code generation system. [Figure 2A] FIG. 2 illustrates the operation of the code generation system. [Figure 2B] FIG. 2 illustrates an example of a computer program generated by the system. [Diagram 3] FIG. 1 is a flow diagram of an exemplary process for generating computer code. [Figure 4] FIG. 1 is a flow diagram of an exemplary process for training a generative neural network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Like reference numbers and designations in the various drawings indicate like elements.

[0025] 1 is a diagram of an example code generation system 100. Code generation system 100 is an example of a system in which the systems, components, and techniques described below may be implemented, implemented as a computer program on one or more computers at one or more locations.

[0026] System 100 generates computer program code in response to received requests using neural network 110. Specifically, in response to a given request, system 100 generates one or more synthesized computer programs 150 that, when executed, perform the computer programming tasks specified in the request.

[0027] In particular, the system 100 receives description data 102 that describes a computer programming task, ie, describes the type of output that should be produced by executing a computer program against inputs for the task.

[0028] Generally, the descriptive data 102 includes natural language text that describes the intended functionality of a computer program to be generated by the system 100. For example, the system 100 can receive the natural language text as input from a user and generate the descriptive data 102 from the natural language text.

[0029] Natural language text may be represented as a sequence of text tokens, eg, tokens that represent one or more of characters, bytes, wordpieces, words, punctuation marks, and the like.

[0030] For example, the system 100 can generate a sequence from a natural language input by applying a tokenizer, such as the SentencePiece tokenizer or another tokenizer, to split the natural language input into tokens from a vocabulary.

[0031] For example, system 100 can train a tokenizer on data that includes a mixture of natural language text sequences and computer code segments in various programming languages, e.g., Python, C++, C#, Java, Ruby, PHP, etc., to ensure that the tokenizer can effectively tokenize programs from various programming languages ​​and natural language descriptions of tasks received as input by system 100.

[0032] The system 100 also obtains one or more sets of inputs 104 for the computer programming task. Each set of inputs 104 satisfies the requirements for an input to the computer programming task, and includes, for example, a respective value for each input variable required to perform the task. As a simple example, when the task described by the description 102 requires sorting a set of input numbers, each input will include a set of numbers to be sorted.

[0033] In particular, the system 100 may obtain (i) a set of filtering inputs 106, (ii) a set of clustering inputs 108, or (iii) both.

[0034] The set of filtered inputs 106 includes a set of inputs and, for each input, a respective expected output generated by executing a computer programming task on the input. The filtered input-expected output pairs may be received by the system 100 along with the descriptive data, e.g., as input from a user.

[0035] The set of clustering inputs 108 includes a set of inputs for a computer programming task, but does not require that there be any expected outputs for the inputs.

[0036] As described in more detail below, in some implementations, the system 100 generates some or all of the clustering inputs 108 from the descriptive data 102 using a test input generation neural network.

[0037] The system 100 then uses the descriptive data 102 to generate a number of candidate computer programs.

[0038] In particular, the system 100 can generate candidate output programs by sampling a number of output sequences from a set of one or more generative neural networks 110.

[0039] Each generative neural network 110 in the set is configured to receive an input sequence including a plurality of text tokens in the descriptive data 102 and to process the input sequence to generate an output sequence including a plurality of computer code tokens that specify candidate computer programs for performing the computer programming task described by the descriptive data.

[0040] Each computer code token is selected from a vocabulary of tokens that represent code symbols in one or more computer programming languages, e.g., Python, C++, C#, Java, Ruby, PHP, etc. For example, the vocabulary of tokens may be the same as the input vocabulary of tokens described above, e.g., the vocabulary employed by the tokenizer described above.

[0041] Neural network 110 may have any suitable neural network architecture that enables the model to map an input sequence of tokens from a vocabulary to an output sequence of tokens from the vocabulary.

[0042] For example, each neural network 110 may have an encoder-decoder architecture, e.g., an architecture with a transformer encoder that applies self-attention across an input sequence to generate an encoded representation of the input sequence, and a transformer decoder that applies cross-attention to the output of the transformer.

[0043] In particular, each neural network 110 may be an autoregressive neural network that autoregressively generates an output sequence of tokens by generating each particular token in the output sequence conditional on a current input sequence, which includes (i) the input sequence followed by (ii) any text tokens that precede the particular text token in the output sequence.

[0044] More specifically, to generate a particular token, the neural network 110 may process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each token in the vocabulary of text tokens. For example, the penultimate layer of the neural network 110 may generate a score ("logit") for each token in the vocabulary, and the final layer of the neural network 110 may be a softmax layer that maps the scores for the tokens to a probability distribution.

[0045] The neural network 110 can then use the score distribution to select a text token from the vocabulary as the particular text token. For example, the neural network 110 can greedily select the highest scoring token or can sample the token from the distribution using, for example, top-k sampling, nucleus sampling, or another sampling technique.

[0046] In general, because neural network 110 is autoregressive, by sampling from a distribution at each time step, system 100 can generate multiple different candidate output sequences in response to the same input using the same neural network 110. That is, by sampling tokens rather than selecting only the highest scoring token, system 100 introduces stochasticity into the generation process and can generate different sequences by processing the same input multiple times, either in parallel or serially.

[0047] Examples of autoregressive transformer architectures that may be used for the neural network 110 include Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, “Towards a human-like open-domain chatbot,” CoRR, abs / 2001.09977, 2020; and Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, "Attention Is All You Need," arXiv preprint arXiv:1706.03762, 2017.

[0048] In some implementations, the architecture of the generative neural network 110 may be modified compared to that described above to facilitate efficient sampling. For example, the generative neural network 110 may employ multi-query attention instead of multi-head attention. In multi-query attention, each attention head in the encoder, decoder, or both has a different query but shares keys and values. Because keys and values ​​are shared, sampling efficiency is improved and the number of parameters of the neural network 110 is reduced. In particular, memory usage and cache update costs, each of which represents a potential bottleneck during sampling, may be reduced through the use of multi-query attention.

[0049] Table 1 shows some example hyper-parameters of an encoder-decoder transformer with multi-query attention or with multi-head attention that may be used for neural network 110.

[0050] [Table 1]

[0051] This table shows the total number of parameters in the model, n params , the hidden dimension of the transformer block d model We list the number of query and key-value (KV) heads, the number of transformer blocks in the encoder (enc) and decoder (dec), the training batch size, the number of gradient update steps, and the number of total training tokens. All of these example configurations share a head size of 128 and a feed-forward fan-out ratio of 6.

[0052] When there are multiple neural networks 110 in the set of generative neural networks 110, each generative neural network 110 has a combination of values ​​for a set of one or more properties that differs from each other neural network 110 in the set. For example, the properties may include one or more of: (i) the training data used to train the generative neural network 110, (ii) the initialized values ​​of the parameters of the generative neural network 110 prior to training the generative neural network 110, (iii) the number of parameters of the generative neural network 110, or (iv) a sampling temperature for sampling the output sequence from the generative neural network 110. Sampling temperatures are described in more detail below.

[0053] By ensuring that each neural network 110 in the set has a different combination of values ​​for this set of properties, the system 100 can ensure that the neural networks 110 can be used to generate a diverse set of outputs for a given input.

[0054] For each candidate computer program in the subset of candidate computer programs generated using neural network 110, and for each input in the one or more sets of inputs 104, system 100 executes the candidate computer program against the input to generate an output.

[0055] The system 100 may execute computer programs in any of a variety of ways.

[0056] For example, the system 100 may use a suitable compiler to compile a given candidate computer program into an executable file and then cause a computer to execute the machine instructions in the executable file.

[0057] As another example, the system 100 may execute a given candidate computer program using a suitable interpreter that translates and executes the code in the computer program.

[0058] If a candidate program fails to compile or cannot be interpreted by the interpreter, the system 100 may remove the candidate program from consideration.

[0059] The system 100 then selects one or more computer programs from the candidate computer programs as a composed computer program 150 for performing the computer programming task based at least in part on outputs generated by executing the candidate computer programs in the subset against inputs in the one or more sets of inputs 104.

[0060] That is, the system uses one or more sets of inputs 104 to “filter” a set of candidate computer programs to produce one or more synthesized computer programs 150 .

[0061] This filtering is described in more detail below with reference to FIG. 2A.

[0062] Once the system 100 has selected one or more computer programs 150, the system 100 may use the selected programs for any of a variety of tasks.

[0063] For example, system 100 may provide source code for each selected computer program 150 for presentation to a user in a user interface to enable, for example, the user to select one of the selected computer programs for inclusion in a larger computer program being programmed by the user, or to select one of the selected computer programs for modification before it is used to perform a task.

[0064] As another example, the system 100 can directly use the selected computer program to perform the task. For example, the system 100 can receive new input for a computer programming task, e.g., submitted by a user through a user interface, provided by another computer program running on the same computer or a remote computer, or received through an application programming interface (API) provided by the system 100, and execute one or more of the composed computer programs 150 against the new input to generate output for the computer programming task for the new input. The system 100 can then provide one or more of the generated outputs as output for the task. For example, when there is only a single selected computer program 150 or when each computer program 150 generates the same output, the system 100 can provide a single output as output for the task. When there are multiple selected computer programs and the programs generate different outputs, the system 100 can provide the output generated by the most number of programs as output for the task.

[0065] Prior to using neural network 110 to generate a synthesized computer program, system 100 or another training system trains neural network 110 with training data. In some cases, as described in more detail below, the training system first pre-trains neural network 110 with computer code sequences and then fine-tunes neural network 110 for code generation purposes, which require generating code from a task description.

[0066] FIG. 2A illustrates operations performed by system 100 to generate one or more composite computer programs 250.

[0067] The system 100 receives task description data 102 and then performs large-scale sampling 210 on the task description data 102 using a generative neural network 110 to generate a large set of candidate computer programs 212. For example, the system 100 may continue sampling 210 until a computational budget allocated for sampling is exhausted or until a threshold number of samples have been generated.

[0068] In other words, the system 100 uses the task description data and the generative neural network 110 to generate a large set of different candidate computer programs 212 .

[0069] That is, the system 100 generates one or more input sequences, each including a text token in the descriptive data 102, and for each generated input sequence, samples a respective set of output sequences from each of one or more of the generative neural networks 110 while the generative neural networks 110 are conditioned on the generated input sequence.

[0070] The system 100 may use any suitable technique for sampling from the autoregressive model samples from a given generative neural network 110 while the neural network 110 is conditioned on the descriptive data, such as nuclear sampling, reduced temperature sampling, greedy sampling, etc.

[0071] In some implementations, each input sequence includes only text tokens in the descriptive data 102, and the system 100 relies on the probabilistic nature of sampling from a given neural network 110, different outputs produced by different neural networks 110, or both, to generate diverse candidates.

[0072] In some other implementations, each input sequence also includes "metadata" tokens that characterize desired properties of the output sequence. For example, these metadata tokens may be inserted before or after the descriptive data 102 in the input sequence and may be formatted according to a specified format.

[0073] Metadata tokens may include tokens that specify any of a variety of properties of a computer program.

[0074] As an example, the metadata tokens may include one or more tokens that identify the programming language that the computer program will be written in. For example, these tokens may be formatted as the token "LANGUAGE IS" followed by the name of the programming language, e.g., "Python" or "C++".

[0075] As another example, the metadata tokens may include one or more tokens that indicate whether the output sequence produced by the generative neural network should specify a correct or incorrect answer for a computer programming task. For example, these tokens may be formatted as "CORRECT SOLUTION" when a correct answer is desired and as "INCORRECT SOLUTION" when an incorrect answer is desired.

[0076] As another example, the metadata tokens may include one or more tags that identify algorithms, e.g., "Divide and Conquer," "Dynamic Programming," and "Data Structures," respectively, that should be implemented by the output sequence generated by the generative neural network. For example, these tokens may be formatted as "TAGS:" followed by an identifier for any algorithms that should be implemented. For some inputs, the input sequence may include zero tags, e.g., when the algorithm is well adapted to the current task.

[0077] As another example, the metadata tokens may include one or more tokens that identify a difficulty rating for a task, e.g., as a numerical rating between "Divide and Conquer," "Dynamic Programming," and "Data Structures" on a predefined rating scale. For example, these tokens may be formatted as "RATING:" followed by a numerical value.

[0078] During training, when generating input sequences, the system 100 generates these metadata tokens to match the actual properties of the output sequence that is provided as a target for the current input sequence, i.e., the system includes metadata that indicates whether the programs in the training examples are correct answers to the descriptions represented by the input sequences in the training examples, the actual difficulty ratings for the tasks described by the input sequences, the actual computer languages ​​in which the programs in the training examples are written, the actual algorithms implemented within the programs in the training examples or that have otherwise been determined to be useful for the tasks described in the input sequences, etc.

[0079] However, at sampling time, ie, after the neural network 110 has been trained, the system 100 can use “metadata conditioning” to improve the diversity and accuracy of the candidate programs 212 .

[0080] That is, at sampling time, the final computer program has not yet been generated, so system 100 does not have access to the generated final computer program, and therefore does not have access to the actual properties of the final computer program.

[0081] Instead, the system 100 uses “metadata conditioning” to select, for each generated input sequence, values ​​for properties that will result in metadata that produces high quality and diverse candidate programs 212 .

[0082] As an example, when the metadata tokens include one or more tokens that identify a programming language in which a computer program is to be written, system 100 may sample a programming language from a distribution across the set of programming languages ​​each time a current input sequence is generated and include one or more tokens that identify the programming language in the current input sequence.

[0083] As another example, when the metadata tokens include one or more tokens indicating whether the output sequence generated by the generative neural network should specify a correct answer or an incorrect answer for the computer programming task, the system 100 may include, each time a current input sequence is being generated, one or more tokens in the current input sequence indicating that the output sequence generated by the generative neural network should specify a correct answer for the computer programming task, i.e., because candidate programs 212 that are not the correct answer are not useful at the sampling time.

[0084] As another example, when the metadata token includes one or more tags, each including one or more tokens that identify an algorithm to be implemented by the output sequence generated by the generative neural network, the system 100 can sample one or more tags from a distribution over the set of tags each time a current input sequence is generated and include the sampled tags in the current input sequence.

[0085] As another example, when the metadata tokens include one or more tokens that identify a difficulty rating for a task, the system 100 may select a difficulty rating for the computer programming task and include in the current input sequence one or more tokens that specify the selected difficulty rating.

[0086] For example, after training the generative neural network 110, the system 100 can either select a fixed difficulty rating that is the same for all computer programming tasks, or sample a difficulty rating from a distribution over the set of difficulty ratings and use the sample difficulty rating as the selected difficulty rating.

[0087] Thus, to generate candidate programs 212, i.e., to sample a number of output sequences representative of candidate programs 212, system 100 repeatedly performs the following operations: (i) generating a current input sequence from descriptive data 102; and (ii) sampling one or more output sequences by processing the current input sequence using one of the generative neural networks 110 in the set. When "metadata conditioning" is employed, system 100 performs the "metadata conditioning" as part of generating the current input sequence.

[0088] In some implementations, the system 100 repeatedly performs multiple instances of generating candidates and sampling output sequences in parallel. For example, the system can perform the generating and sampling in parallel using multiple parallel processing devices, such as CPUs, GPUs, TPUs, or other ASICs, FPGAs, etc. As a particular example, the system 100 can deploy multiple instances of each of one or more generative neural networks 110 across those devices. For example, the system 100 can deploy each instance on a different device, or deploy the instances across multiple devices using model parallelism techniques. The system 100 can then sample candidates for each instance as described above using the instance in parallel with each other instance.

[0089] In some of these implementations, the system 100 may also perform execution of the candidate programs 212 in parallel across multiple devices, for example, by executing each candidate generated by a given instance on the device on which the instance is deployed.

[0090] In the example of FIG. 2A, the system 100 receives a set of inputs including a filtering input 106 and a clustering input 108 .

[0091] The system then performs filtering and clustering 220 on the set of candidate computer programs 212 to select one or more combined computer programs 250.

[0092] Generally, as part of performing filtering and clustering 220, for each candidate computer program in the subset of candidate computer programs 212, and for each input in the one or more sets of inputs 104, system 100 executes the candidate computer program against the input to generate an output. As explained above, if a given candidate fails to compile or cannot be interpreted, system 100 removes the candidate from consideration.

[0093] In some cases, system 100 may maintain data that specifies criteria for latency or resource consumption, or both. If the execution of any given candidate lacks either of the criteria, system 100 removes the candidate from consideration. Thus, system 100 may ensure that synthesized program 250 remains resource efficient.

[0094] The system 100 then selects one or more computer programs from the candidate computer programs 212 as a composed computer program 250 for performing the computer programming task based at least in part on the output generated by executing the candidate computer programs in the subset on the input.

[0095] In particular, when one or more sets include a filtered input 106, the subset (of candidate computer programs 212 to be executed on the input) is generally not a proper subset, i.e., the subset includes all of the plurality of candidate computer programs 212.

[0096] The system can then determine, for each candidate computer program, and for each of the filtering inputs 106, whether the output produced by executing the candidate computer program against the filtering inputs 106 matches the respective expected output for the filtering inputs 106.

[0097] The system 100 may then select, for each filtering input 106, only those candidate computer programs that, when executed against the filtering input 106, produced outputs that matched the respective expected outputs for the filtering input 106 as an initial set of computer programs.

[0098] When one or more sets also include clustering inputs 108, the system 100 may then cluster the candidate computer programs in the initial set into multiple clusters based on outputs generated by running the candidate computer programs on the clustering inputs. The system may use any suitable clustering technique to cluster the candidate computer programs based on the outputs generated by the clustering inputs. For example, the system 100 may group two candidates into the same cluster only if the two candidates generate matching outputs for at least a threshold percentage of the clustering inputs.

[0099] The system 100 may then select one or more respective candidate computer programs from each of one or more of the clusters as the combined computer program 250.

[0100] As a particular example, the system 100 may select one or more respective computer programs from each of a predetermined number of the largest clusters of the plurality of clusters as the composite computer program 250, where the size of the cluster is determined based on the number of candidates in the cluster. Selecting the composite computer program 250 in this manner takes advantage of the fact that there are many ways in which a solution may be incorrect, but that correct answers tend to behave in a similar way and are therefore grouped into larger clusters.

[0101] In other examples, the system 100 receives only the filtering input 106. In these cases, the system 100 may select one or more synthesized programs 250 from the initial set, for example, by selecting a fixed number randomly or based on one or more heuristics, such as the length of the programs.

[0102] In yet other examples, the system 100 receives only clustering input 108. In these cases, the system 100 can cluster all of the candidate programs as described above, rather than clustering only those programs in the initial set.

[0103] In some implementations, the system 100 receives all of the clustering inputs 108 as input from a user. In some other implementations, instead of or in addition to receiving the clustering inputs 108 as input, the system 100 can generate one or more "composite" clustering inputs 108.

[0104] Specifically, system 100 may process one or more description sequences, each including a plurality of text tokens from description data 102, using a test input generation neural network configured to process each description sequence to generate as output one or more test inputs for the computer programming task described by the description sequence. Although these synthetic inputs are not guaranteed to be valid, incomplete or even invalid test inputs may still be useful for grouping candidate programs, especially when the task has complex constraints.

[0105] Training the test input generation neural network is described below with reference to FIG.

[0106] FIG. 2B shows an example of a computer program 290 generated by the system.

[0107] 2B, a user provides description data 252 that describes a problem, i.e., a task, in natural language. The user also provides a set of filtering inputs 254 and expected outputs for each filtering input.

[0108] In the example of FIG. 2B, each filtering input 254 is a pair of strings, and the expected output for the pair of strings indicates whether one string can be obtained from the other string by pressing "backspace" instead of typing one or more of the characters in the first string. The system 100 uses the neural network 110 to process an input sequence including description data 252, possibly including filtering inputs 254 and notes 256 on the filtering inputs, to generate candidate programs, and then filters those candidates using at least the filtering inputs 254 to arrive at the synthesized computer program 290. That is, in some cases, the filtering inputs 254 are considered part of the description data 252 in addition to being used to filter the candidate programs. As can be seen from FIG. 2B, the synthesized computer program 290 performs exactly the task described in the description data 252.

[0109] 3 is a flow diagram of an exemplary process 300 for generating one or more synthesized computer programs for performing tasks. For convenience, the process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a code generation system suitably programmed in accordance with this specification, such as the code generation system 100 shown in FIG. 1, may perform the process 300.

[0110] The system receives description data describing a computer programming task, step 302. As explained above, the description data includes a number of text tokens that describe the task.

[0111] The system receives a first set of inputs for a computer programming task (step 304).

[0112] The system generates a plurality of candidate computer programs by sampling a plurality of output sequences from a set of one or more generative neural networks (step 306). As described above, each generative neural network in the set is configured to receive an input sequence including a plurality of text tokens in the descriptive data and to process the input sequence to generate an output sequence including a plurality of computer code tokens that designate candidate computer programs for performing the computer programming task described by the descriptive data.

[0113] For each candidate computer program in the subset of candidate computer programs, and for each input in the first set, the system executes the candidate computer program against the input to generate an output (step 308).

[0114] The system selects one or more computer programs from the candidate computer programs as a composed computer program for performing the computer programming task based at least in part on outputs produced by executing the candidate computer programs in the subset on inputs in the first set of inputs (step 310). That is, the system "filters" the candidate computer programs using the first set of inputs to select one or more composed computer programs.

[0115] As explained above, prior to using the generative neural network to generate computer code, the system or a different training system trains the generative neural network.

[0116] For example, the system may pre-train each generative neural network for language modeling purposes based on a first set of training data, and then fine-tune the generative neural network for code generation purposes based on a second set of training data, as described in more detail below with reference to FIG.

[0117] 4 is a flow diagram of an exemplary process 400 for training a generative neural network. For convenience, the process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a code generation system suitably programmed in accordance with this specification, such as the code generation system 100 shown in FIG. 1, can perform the process 400.

[0118] The system obtains a first set of training data (step 402).

[0119] The first set of training data includes a set of sequences each representing a segment of computer code.

[0120] For example, the system may obtain a repository of computer code segments, e.g., computer code files, or portions of computer code files, that include computer code segments written in multiple computer programming languages. The system may then generate a first set of training data by "flattening" each computer code segment into a sequence, e.g., by removing whitespace or replacing whitespace with designated tokens.

[0121] Thus, the first set of training data includes sequences representing computer code, but does not include any task descriptions that describe the tasks accomplished by a given computer code segment (apart from any task information that will be embedded in comments within the computer code segment).

[0122] The system pre-trains a generative neural network on a first set of training data (step 404).

[0123] Specifically, the system trains a generative neural network based on the first set of training data for language modeling purposes that require the generative neural network to predict a token that follows a given subsequence of tokens in a given sequence, given a subsequence of tokens from a given sequence in the first set of training data.

[0124] For example, when the neural network has an encoder-decoder architecture as described above, the system can sample the pivot points in each sequence. The system can then generate training examples that include the tokens before the pivot points in the sequence as inputs and the tokens after the pivot points in the sequence as target outputs.

[0125] The language modeling objective may then require the neural network to predict a target output in each training example by processing input sequences in the training examples.

[0126] For example, the language modeling objective may be the cross-entropy next-token prediction loss for a decoder given the coded representations of input sequences in the training examples produced by the encoder.

[0127] In some cases, the system may include one or more auxiliary losses as part of pre-training.

[0128] For example, the system can incorporate a masked language modeling loss to the encoder's predictions during pre-training. Masked language modeling loss is described in more detail in J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018.

[0129] Although the pre-training dataset does not contain task descriptions and therefore does not directly train the generative neural network to perform the code generation task, the pre-training stage can help the generative neural network learn a good representation of the code and generate the code fluently. As a result, the model can reasonably represent the human coding space, thereby significantly reducing the problem search space.

[0130] The system obtains a second training data set for fine-tuning the generative neural network (step 406).

[0131] The second training data set includes a plurality of training examples, each of which includes (i) a training input sequence representing a computer programming task, and (ii) a training output sequence representing a computer program.

[0132] As described above, in some implementations, after training, the inputs to a generative neural network include "metadata" tokens that each specify properties of the output sequence that will be generated by the neural network.

[0133] In these implementations, the system also augments each training input sequence with metadata tokens that specify corresponding actual properties of the computer program represented by the output sequences in the training examples.

[0134] In some of these implementations, the system can leverage this metadata to enable the system to incorporate incorrect computer programs into the training.

[0135] Specifically, the training examples may include both a) a first subset of training examples in which the training output sequence in the training example is a correct answer to the computer programming task represented by the training input sequence in the training example, and b) a second set of training examples in which the training output sequence in the training example is not a correct answer to the computer programming task represented by the training input sequence in the training example. For each training example in a), the system includes a metadata token that indicates a correct answer, but for each training example in b), the system includes a metadata token that indicates an incorrect answer. This provides an additional training signal and enables the use of data that might otherwise mislead the model.

[0136] The system trains a generating neural network of the code generating neural network (step 408).

[0137] That is, the system trains the generative neural network with the goal of encouraging it to generate computer programs that accurately perform the tasks specified by the input sequences in each training example.

[0138] In some implementations, the system employs "tempering" during training. Tempering is a regularization technique that artificially makes the token probability distributions generated by a generative neural network smoother or sharper at training time by dividing the neural network's output logits, i.e., the scores generated by the penultimate layer of the neural network, by a scalar temperature T before the softmax layer. When tempering is employed, the system uses a temperature T between 0 and 1, excluding the numbers at both ends. By using a temperature within this range, overfitting on the fine-tuning dataset can be avoided by making the training distribution sharper and, therefore, the inference distribution smoother. For example, the system can set T equal to .1, .2, or .4.

[0139] In these implementations, at sampling times, the system divides the logit by another temperature T' between 0 and 1, excluding the extreme numbers. For example, the system can determine T' by tuning the value on a validation set after training.

[0140] In some implementations, the system uses maximum likelihood as the code generation objective.

[0141] In some other implementations, the system uses the GOLD objective as the code generation objective. GOLD is an offline RL algorithm that adds off-policy importance weights to a standard maximum likelihood objective gradient. Using the GOLD objective can take into account the fact that code generation based on a task description is essentially a one-of-many task, i.e., each task problem allows for many distinct solutions that depend on the algorithm choice, implementation, etc. The standard maximum likelihood objective minimizes loss by placing some weight on each solution in the training set (like recall), whereas the goal of the system is to arrive at a single correct solution that is within a budget (like precision). GOLD can take this into account by adding off-policy importance weights, which allows the model to both learn from tokens that it has already assigned a high likelihood to and ignore tokens that are not in its distribution. In this way, the model can focus on precision over recall and increase its chances of getting at least one correct sample. More specifically, the gradient of the GOLD objective satisfies:

[0142]

number

[0143] where θ is the parameter of the generative neural network, the solution tokens are the tokens in the output sequence of the training examples, and P θ (s) is the probability assigned to token s by the generative neural network.

[0144] In some cases, to reduce instability during training, the system may set the importance weight (s) above to max(P θ (s) α,β), with α = 12 and β = 0.05.

[0145] In some of these implementations, the objective is GOLD with tempering, in which the system can first divide the logits by the temperature before computing the loss, so that both the log loss term and the importance weights use a tempered distribution.

[0146] In some cases, the system may include one or more auxiliary losses as part of the fine tuning.

[0147] For example, the system can incorporate a masked language modeling loss to the encoder's predictions during fine-tuning. Masked language modeling losses are described in more detail in J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018.

[0148] As another example, the system can incorporate a value prediction auxiliary task into the training of the decoder. In this example, the final layer token representations before projecting to logits are also used by a prediction head, e.g., a small transformer, to classify the generated computer program as correct or incorrect.

[0149] When the test input generating neural network is used at inference time, the test input generating neural network can also have an encoder-decoder architecture like the generative neural network. Specifically, the system can train the test input generating neural network after step 404 starting from the pre-trained generative neural network, but using a different fine-tuning objective. Specifically, the fine-tuning objective can be to predict the test input from the problem description using the known test input as the target output. For example, the objective can be a maximum log-likelihood objective.

[0150] As explained above, the system can make any of several adjustments to the training and sampling process to improve the system's performance. Table 2 shows the effect of some of these adjustments on the performance of the system on a task that requires the submission of 10 synthesized programs for each natural language description and is determined to perform successfully when one or more of the 10 programs perform the task successfully.

[0151] [Table 2]

[0152] Each row in Table 2 shows the solve rate of the system using the adjustment named in that row and all of the adjustments named in all rows above it, with the numbers in parentheses being the 95% confidence interval for the solve rate. The "solve rate" is the proportion of inputs for the task described above for which the system produced a successful program. Row 2, "No Augmentations," corresponds to using a fine-tuned neural network with a standard next token prediction loss (no GOLD or tempering, and no masked language modeling or value prediction auxiliary losses in training, and no clustering at sampling time). Meanwhile, the last row corresponds to using the "No Augmentations" system, but with masked language modeling, tempering, random tags and ratings (using "metadata conditioning" with randomly selected tags and ratings, as described above), value prediction, GOLD, and clustering at sampling time. As can be seen from Table 2, each adjustment improves the performance of the system compared to a system that does not include that adjustment.

[0153] This specification uses the term "configured" in connection with systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action during operation. A computer program or programs configured to perform a particular operation or action means that the program or programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0154] Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or a combination of one or more of them, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein may be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or sequential access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0155] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In some cases, an apparatus may include, in addition to hardware, code that creates an execution environment for a computer program, e.g., code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0156] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any type of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may correspond to a file in a file system, but need not. A program may be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program may be deployed to run on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0157] As used herein, the term "database" is used broadly to refer to any collection of data, which need not be structured in any particular way, or at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.

[0158] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine, and in other cases, multiple engines may be installed and running on the same computer or computers.

[0159] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., FPGAs or ASICs, or a combination of special purpose logic circuitry and one or more programmed computers.

[0160] A computer suitable for executing a computer program may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. In general, the central processing unit will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated in special-purpose logic circuitry. In general, a computer will also include one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or be operatively coupled to receive data therefrom, or transfer data thereto, or both. However, a computer need not have such devices. Moreover, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few examples.

[0161] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks.

[0162] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may be used to provide for user interaction as well, e.g., feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, speech input, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to the web browser in response to a request received from a web browser on the user's device. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a response message from the user in return.

[0163] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling common and computationally intensive parts of, e.g., machine learning training or production, e.g., inference, workloads.

[0164] The machine learning model may be implemented and deployed using a machine learning framework, for example, the TensorFlow framework.

[0165] An embodiment of the subject matter described herein may be implemented in a computing system that includes back-end components, e.g., as a data server, or includes middleware components, e.g., an application server, or includes front-end components, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user may interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0166] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device, e.g., to display data to and receive user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., results of user interaction, may be received at the server from the device.

[0167] Although the present specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Some features described in the present specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in some combinations, and may even be initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination, or a variation of a subcombination.

[0168] Similarly, although operations are illustrated in the figures and claimed in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequence illustrated, or that all of the illustrated operations be performed to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0169] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes illustrated in the accompanying figures do not necessarily require the particular order illustrated, or sequence, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0170] 100 Code generation system, system 102 Descriptive data, description, task descriptive data 104 Input 106, 254 Filtering Input 108 Clustering input, "composite" clustering input 110 Neural Networks, Generative Neural Networks 150, 290 Synthesized computer programs, computer programs 210 Large-scale sampling, sampling 212 Candidate computer program, candidate program 250 Synthesized computer programs, Synthesized programs 252 Descriptive Data 256 Note

Claims

1. A method implemented by one or more computers, comprising: receiving description data describing a computer programming task, said description data comprising a plurality of text tokens; receiving a first set of inputs for the computer programming task; generating a plurality of candidate computer programs by sampling a plurality of output sequences from a set of one or more generative neural networks, each generative neural network in the set being configured to receive an input sequence including the plurality of text tokens in the descriptive data and to process the input sequence to generate an output sequence comprising a plurality of computer code tokens that designate candidate computer programs for performing the computer programming task described by the descriptive data; for each candidate computer program in the subset of candidate computer programs and for each input in the first set, executing the candidate computer programs on the input to generate output; selecting one or more computer programs from the candidate computer programs as a composed computer program for performing the computer programming task based at least in part on the outputs generated by executing the candidate computer programs in the subset on the inputs in the first set of inputs; The method includes:

2. receiving new input for the computer programming task; executing one or more of the synthesized computer programs on the new input to generate output for the computer programming task for the new input; The method of claim 1, further comprising:

3. The method further comprising: receiving, for each input in the first set, a respective expected output generated by executing the computer programming task on the input; and wherein the selecting step further comprises: For each candidate computer program in the subset and for each input in the first set, determining whether the outputs produced by executing the candidate computer programs on the inputs match the respective expected outputs for the inputs; selecting, for each input in said first set, as an initial set of computer programs only those candidate computer programs which, when executed on said input, produced an output that was consistent with said respective expected output for said input; Including, 3. The method of claim 1 or 2, wherein selecting one or more computer programs from the candidate computer programs as a composed computer program for performing the computer programming task based at least in part on the outputs generated by executing the candidate computer programs in the subset on the inputs in the first set of inputs comprises selecting one or more respective computer programs from the initial set of computer programs as a composed computer program.

4. 4. The method of claim 1, wherein the subset of candidate computer programs comprises all of the plurality of candidate computer programs.

5. The selecting step comprises: clustering the candidate computer programs in the subset into a plurality of clusters based on the outputs generated by executing the candidate computer programs on the inputs in the first set; selecting one or more respective candidate computer programs from each of one or more of said clusters as a combined computer program; 3. The method of claim 1 or 2, comprising:

6. selecting one or more respective computer programs from each of one or more of the clusters as a combined computer program, selecting one or more respective computer programs from each of a predetermined number of largest clusters of said plurality of clusters as a combined computer program.

6. The method of claim 5, comprising:

7. obtaining the first set of inputs for the computer programming task, processing one or more description sequences, each including the plurality of text tokens from the description data, using a test input generation neural network, the test input generation neural network being configured to process each description sequence to generate as output one or more test inputs for the computer programming task described by the description sequence.

7. The method of claim 5 or 6, comprising:

8. 8. The method of claim 5, wherein the subset of candidate computer programs includes only those candidate computer programs that were not filtered out from the plurality of candidate computer programs based on whether, for each candidate computer program, and for each input in a second set of inputs, the output generated by executing the candidate computer program on the input matched a respective expected output for the input.

9. 9. The method of claim 1, wherein the set of one or more generative neural networks includes only a single generative neural network.

10. 9. The method of claim 1, wherein the set of one or more generative neural networks comprises a plurality of generative neural networks, each generative neural network in the set having a combination of values ​​for a set of one or more properties that differs from each other neural network in the set, the properties including one or more of: (i) training data used to train the generative neural network; (ii) a number of parameters of the generative neural network; or (iii) a sampling temperature for sampling an output sequence from the generative neural network.

11. One or more of the generative neural networks in the set: an encoder neural network configured to process the input sequence to generate an encoded representation of the input sequence; a decoder neural network configured to process the encoded representation to generate the output sequence; and 11. The method of any one of claims 1 to 10, comprising:

12. 12. The method of claim 11 , wherein the encoder neural network is a Transformer Encoder that applies self-attention across the input sequence, and the decoder neural network is an Autoregressive Transformer Decoder that applies cross-attention to the encoded representations.

13. generating a plurality of candidate computer programs by sampling a plurality of output sequences from a set of one or more generative neural networks, generating a current input sequence from the descriptive data; sampling one or more output sequences by processing the current input sequence using one of the generative neural networks in the set; 13. The method of claim 1, comprising repeatedly performing operations including:

14. The method of claim 13 , wherein the operations are performed in parallel across multiple hardware devices.

15. each input sequence comprising the plurality of text tokens from the description data and one or more tokens identifying a programming language, and generating a current input sequence from the description data; Sampling programming languages ​​from a distribution over a set of programming languages; including in the current input sequence one or more tokens that identify the programming language; 15. The method of claim 13 or 14, comprising:

16. each input sequence comprising the plurality of text tokens from the description data and one or more tokens indicating whether the output sequence produced by the generative neural network should specify a correct answer or an incorrect answer for the computer programming task, and generating a current input sequence from the description data; including one or more tokens indicating that the output sequence generated by the generative neural network should specify a correct answer for the computer programming task.

16. The method of any one of claims 13 to 15, comprising:

17. generating a current input sequence from the descriptive data, each input sequence comprising the plurality of text tokens from the descriptive data and zero or more tags each comprising one or more tokens and identifying an algorithm to be implemented by the output sequence generated by the generative neural network; Sampling one or more tags from a distribution over a set of tags; including the sampled tag in the current input sequence; 17. The method of any one of claims 13 to 16, comprising:

18. each input sequence comprising the plurality of text tokens from the description data and one or more tokens specifying a difficulty rating for the computer programming task, and generating a current input sequence from the description data comprises: selecting a difficulty rating for the computer programming task; including, in the current input sequence, one or more tokens that specify the selected difficulty rating; 18. The method of any one of claims 13 to 17, comprising:

19. selecting a difficulty rating for the computer programming task, selecting a fixed difficulty rating that is the same for all computer programming tasks after training the generative neural network.

20. The method of claim 18, comprising:

20. selecting a difficulty rating for the computer programming task, Sampling difficulty ratings from a distribution over the set of difficulty ratings.

20. The method of claim 18, comprising:

21. 21. The method of any one of claims 1 to 20, wherein each generative neural network is pre-trained for language modeling purposes based on a first set of training data and fine-tuned for code generation purposes based on a second set of training data including a plurality of training examples, each of the training examples comprising (i) training input sequences representing computer programming tasks and (ii) training output sequences representing computer programs.

22. 22. The method of claim 21, wherein the first set of training data comprises a set of sequences each representing a segment of computer code.

23. 23. The method of claim 21 or 22, wherein the plurality of training examples comprises a first subset of training examples in which the training output sequence in the training examples is a correct answer to the computer programming task represented by the training input sequence in the training examples, and a second set of training examples in which the training output sequence in the training examples is not a correct answer to the computer programming task represented by the training input sequence in the training examples.

24. 24. The method according to any one of claims 21 to 23, wherein the code generation objective is GOLD with tempering.

25. 25. The method of claim 1, wherein the text tokens and the computer code tokens are selected from the same vocabulary of tokens.

26. 26. A system comprising one or more computers and one or more storage devices, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of a respective method according to any one of claims 1 to 25.

27. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations of the respective method recited in any one of claims 1 to 25.

Images