Generating natural language outlines of computer code using generative neural networks

Abstract

Methods, systems, and apparatuses, including computer programs encoded on computer storage media, for generating natural language outlines of computer code segments using a generative neural network. In particular, the described techniques include obtaining a code input characterizing a computer code segment and processing an input sequence generated from the code input using a generative neural network to generate a natural language outline. Because the natural language outline partitions the computer code segment into logical sections and includes a respective natural language statement for each logical section that summarizes the logical section in natural language, the described techniques enable efficient computer code generation, updating, and validation.

Description

[0001] GENERATING NATURAL LANGUAGE OUTLINES OF COMPUTER CODE USING GENERATIVE NEURAL NETWORKS

[0002] CROSS-REFERENCE TO RELATED APPLICATION

[0003] This application claims priority to U.S. Application No. 63 / 679,068, filed August 2, 2024. The disclosure of the foregoing application is hereby incorporated by reference in its entirety.

[0004] BACKGROUND

[0005] This specification relates to processing inputs using neural networks.

[0006] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e.. the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current value inputs of a respective set of parameters.

[0007] SUMMARY

[0008] This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates natural language outlines of computer code segments using a generative neural network.

[0009] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0010] Using generative neural networks to generate computer code segments is challenging due to the requirement that generated code segments have high-level logical structures and reflect semantic intents of a user. While generative neural networks can produce code segments that are syntactically valid, often the generated code segments are functionally incorrect, contain logical flaws, or deviate from the user’s intent. This unreliability arises because the generative neural networks often lack a way to guide the generation of output that represents the computer code segment, leading to unpredictable results. Consequently, directly using a generative neural network to generate, update, or validate computer code is often impractical, as the resulting code segment may be flawed, or the inputs for the generative neural network required to properly constrain the generative neural network may be unwieldy and complex. Therefore, there is a need for techniques that can more effectively direct generative neural networks to produce reliable and accurate code segments.

[0011] This specification describes techniques that can address the aforementioned challenges. That is, this specification describes techniques for obtaining code input characterizing a computer code segment, and generating, using a generative neural network to process an input sequence generated from the code input, an output sequence defining a natural language outline of the computer code segment. The natural language outline partitions the computer code segment into logical sections and includes a respective natural language statement for each logical section that summarizes the logical section in natural language. After generating the natural language outline, the described techniques can use the natural language outline for efficient generation, updating, and validation of the computer code segment.

[0012] The described techniques, through processing an input sequence that includes the natural language outline using a generative neural network, enable efficient generation of the code segment by using the natural language outline to guide the generation of the output. In particular, because the natural language outline includes a natural language statement for each logical section, the described techniques can use the generative neural network to generate correct code that is in accordance with the natural language statement for each logical section. That is, by conditioning the generation of the code corresponding to a logical section on the corresponding natural language statement of the logical section, the generative neural network will generate correct code that is in accordance with the natural language statements included in the natural language outline. Therefore, the generated code segment is functionally correct, logically consistent, and does not deviate from the user’s intent. Without the natural language outline, the generative neural network would generate a code segment that deviates from desired qualities in unexpected ways (i.e., the generated code will not function appropriately or follow user expectations, e.g., user expectations defined in the natural language statements of the natural language outline).

[0013] The described techniques, through processing an input sequence that includes a modified natural language outline using a generative neural network, enable efficient generation of an updated code segment. In particular, concise modifications to the natural language outline can be responsible for numerous complex code updates during the generation of an updated code segment. That is, the generative neural network can interpret the semantic intention of the modified natural language outline to generate an updated code segment that reflects the modified natural language outline. Therefore, a user can make simple modifications to the natural language outline to reflect desired changes in the computer code segment, which is much simpler than the user directly modifying the computer code segment. Without the natural language outline, the input sequence to generate the updated code segment would need to be greatly more complex and larger, e.g., the input sequence could require the inclusion of the identification of all places in the code segment to update.

[0014] The described techniques, through processing an input sequence that includes a modified code segment using a generative neural network to generate an updated natural language outline for the modified computer code segment, enable efficient validation of the code segment. In particular, the updated natural language outline can be used to easily verify that the modified code segment includes coherent code modifications. For example, the described techniques can verify that the natural language statements are consistent among each other and the modified lines of code and determine that code modifications are valid when consistency is determined to be present. Without the updated natural language outline, there is not an efficient way to validate the modified code segment is valid. The described techniques thus guide a user through a humanmachine interaction, assisting the user in generating the correct code for an intended program function.

[0015] The described techniques can also improve code understanding because the partitioned logical sections of the natural language outline make it easier for humans to organize, navigate, and search for particular code found in the logical sections. Also, because the natural language statements are easier to understand than computer code, the natural language statements for each logical section that summarizes the logical section helps humans understand the code included in the section.

[0016] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

[0017] According to a first aspect there is a provided a method performed by one or more computers. The method includes obtaining a code input characterizing a computer code segment that includes a plurality of lines of computer code. Then the method includes generating, from the code input, an input sequence to a generative neural network. Afterwards, the method includes processing the input sequence using the generative neural network to generate an output sequence defining a natural language outline of the computer code segment. The natural language outline partitions the computer code segment into logical sections and includes a respective natural language statement for each logical section that summarizes the logical section in natural language.

[0018] In some implementations, the code input includes the computer code segment.

[0019] In some implementations, the input sequence includes the lines of computer code from the computer code segment.

[0020] In some cases, the output sequence includes the respective natural language statements interleaved within the computer code segment.

[0021] In some cases, within the output sequence, positions of the respective natural language statements indicate partitions between the logical sections.

[0022] In some cases, within the output sequence, each respective natural language statement is designated by one or more pre-determined code symbols.

[0023] In some cases, the one or more pre-determined code symbols indicate a beginning of a code comment.

[0024] In some cases, processing the input sequence using the generative neural network includes performing constrained generation to cause the generative neural network to copy lines of code from the input sequence to the output sequence.

[0025] In some cases, the input sequence includes a respective identifier for each of the plurality of lines of code.

[0026] Further in some cases, the output sequence includes, for each natural language statement, a respective identifier of a line of code from the plurality of lines of code to which the natural language statement corresponds.

[0027] In some implementations, the code input includes a natural language description of an intended function of the computer code segment.

[0028] In some implementations, the code input includes a prompt that causes the generative neural network to generate the output sequence that defines the natural language outline.

[0029] In some implementations, the generative neural network is configured to process a sequence of tokens to auto-regressively generate an output sequence of tokens.

[0030] In some implementations, the method further includes providing the natural language outline for presentation to a user on a user device.

[0031] In some cases, the providing the natural language outline for presentation to a user on a user device includes providing the natural language outline for presentation in an integrated development environment (IDE) on the user device. In some cases, the providing the natural language outline for presentation to a user on a user device includes providing the natural language statements for presentation interleaved with the lines of code.

[0032] In some cases, the providing the natural language outline for presentation to a user on a user device includes providing the natural language statements for presentation without the lines of code.

[0033] In some cases, the providing the natural language outline for presentation to a user on a user device includes providing the natural language statements for presentation separate from the lines of code in a user interface. Additionally, each natural language statement is presented in the user interface in association with a respective user interface element that when selected by the user causes the corresponding logical section of computed code to be presented in the user interface.

[0034] In some implementations, the method further includes generating, from the natural language outline, a second input sequence. Then, processing the second input sequence using the generative neural network to generate the computer code segment.

[0035] In some cases, generating, from the natural language outline, a second input sequence includes providing the natural language statements for presentation to the user. Then, receiving a user input specifying one or more modifications to the presented natural language statements to generate a new plurality of natural language statements. Then, generating a second input sequence that includes the new plurality of natural language statements.

[0036] In some implementations, the method further includes receiving one or more user inputs modifying the natural language outline to generate a modified natural language outline. Then, generating, from the modified natural language outline, a third input sequence. Then, processing the third input sequence using the generative neural netw ork to generate an updated code segment.

[0037] In some cases, the third input sequence includes the code segment, and the modified natural language outline.

[0038] In some cases, the one or more user inputs further modify the code segment to generate an initial updated code segment. Additionally, the third input sequence includes the modified natural language outline and the initial updated code segment.

[0039] In some implementations, the method further includes receiving one or more user inputs modifying the computer code segment to generate a modified computer code segment. Then, generating, from the modified computer code segment, a fourth input sequence. Then, processing the fourth input sequence using the generative neural network to generate an updated natural language outline for the modified computer code segment.

[0040] In some cases, the fourth input sequence includes the modified code segment, and the natural language outline.

[0041] In some cases, the one or more user inputs further modify the natural language outline to generate an initial updated natural language outline, and the fourth input sequence includes the initial updated natural language outline and the modified code segment.

[0042] According to a second aspect, there is provided the methods of the first aspect performed by one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method.

[0043] According to a third aspect, there is provided the methods of the first aspect performed by one or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method.

[0044] Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0045] BRIEF DESCRIPTION OF THE DRAWINGS

[0046] FIG. 1 shows a natural language outline generation system.

[0047] FIG. 2 is a flow diagram of an example process for generating an output sequence defining a natural language outline of the computer code segment.

[0048] FIG. 3 is an example of an output sequence defining a natural language outline.

[0049] FIG. 4 is an example of a natural language outline for presentation to a user on a user device.

[0050] FIG. 5 is an example of a modified natural language outline and an updated code segment.

[0051] FIG. 6 is an example of a modified computer code segment and an updated natural language outline for the modified computer code segment.

[0052] FIG. 7 is an example of an application that includes the use of the described techniques.

[0053] FIG. 8 is an example of the performance of the described techniques.

[0054] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0055] FIG. 1 shows a natural language outline generation system 100. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0056] The system 100 generates a natural language outline 110 of a computer code segment 112 using a generative neural network 106.

[0057] The computer code segment 112 can be any appropriate portion of computer code that includes multiple lines of computer code. For example, the computer code segment 112 can be a source code file or a portion of a source code file, e.g.. the portion of a source code file corresponding to a single function. The computer code can be any appropriate type of computer code in any appropriate language. For example, the computer code can be code in a programming language, e.g., Python or C++, code in a markup language, e.g., HTML or XML, code in a scripting language, e.g., lavaScript, and so on.

[0058] FIG. 1 shows the code segment 112 as computer code reflecting a Python function definition.

[0059] A natural language outline 110 of a computer code segment 112 partitions the computer code segment 112 into logical sections and includes a respective natural language statement for each logical section that summarizes the logical section in natural language. Each logical section generally includes a respective disjoint subset of the lines of code within the code segment and is considered a “logical section’’ because the lines of code are related to each other by collectively performing a common process or serving a common purpose.

[0060] In other words, the natural language outline 110 identifies, for a given code segment 112, the logical sections within the code segment 112 and includes, for each identified logical section, a natural language statement that summarizes the logical section in natural language. When generating a natural language outline 110, the system 100 can receive “raw” computer code, i.e.. code that hasn’t been divided into any logical sections, and determine both which logical sections to partition the code into and the natural language statement for each of the logical section. The sections are referred to as “logical” because the lines within each of the sections have been determined by the system to be semantically related (i.e.. the lines are related to each other by collectively performing a common process or serving a common purpose). FIG. 1 shows the natural language outline 110 as the natural language outline for the depicted code segment 112. The respective natural language statements for each of the logical sections of the natural language outline 110 begin with the symbol and these natural language statements are interleaved within the computer code segment of the depicted code segment 112.

[0061] More specifically, the system 100 obtains a code input 102 characterizing a computer code segment 112 that includes a plurality of lines of computer code.

[0062] In some cases, the code segment 112 has already been generated and the code input 102 includes the computer code segment 112. In some other cases, the code segment 112 has not yet been generated and the code input 102 includes a natural language description of the intended function of the computer code segment 112.

[0063] The system 100 generates, from the code input 102, an input sequence 104 to a generative neural network 106.

[0064] The generative neural network 106 can be any appropriate neural network that receives as input a sequence of tokens and processes the sequence of tokens to generate an output sequence of tokens. A 'token’ is data that represents a unit of data, e.g., a text symbol or data of another modality, e.g., a portion of an image, audio signal, or video signal. For example, a ‘token’ can be a one-hot vector or a dense embedding.

[0065] In some cases, the generative neural network 106 is a language model neural network that processes tokens representing text symbols or a multi-modal language model neural network that can process tokens representing text symbols and tokens representing data of one or more other modalities, e.g., image, video, audio, and so on. As a particular example of this, the generative neural network 106 can be an auto-regressive neural network that generates the tokens in the output sequence auto-regressively, i.e.. one after another. One example of such a neural network is a decoder-only Transformer neural network. Other examples include neural networks that belong to the Gemini family of neural networks, the Gemma family of neural networks, and so on.

[0066] Thus, the system 100 generates the input sequence 104 from the code input 102 by representing the code input 102 as tokens that can be processed by the generative neural network 106. Optionally, the system 100 can also include, in the input sequence 104, a prompt that causes the generative neural network 106 to generate a natural language outline 110, e.g., a few-shot prompt, a natural language instruction, or both.

[0067] The system 100 processes the input sequence 104 using the generative neural network 106 to generate an output sequence 108 defining a natural language outline 110 of the computer code segment, i.e., that represents the natural language outline 110 as a sequence of tokens, e.g., from a vocabulary of text symbols. As described above, the natural language outline 110 partitions the computer code segment 112 into logical sections and includes a respective natural language statement for each logical section that summarizes the logical section in natural language.

[0068] Once generated, the system 100 can provide the natural language outline 110 for presentation to a user of a user device. In some cases, the system 100 can provide the outline 110 for presentation separate from the code segment, e g., without presenting the code segment 112 or in a separate portion of a user interface from the code segment 112. In some other cases, the system 100 can provide the natural language outline 110 for presentation interleaved with the code segment 112 (as is shown for the natural language outline 110 in FIG. 1).

[0069] In some cases, after generation, the system 100 can allow users to modify the natural language outline 110, the code segment 112, or both. Such modifications can trigger the system 100 to generate an updated natural language outline, an updated code segment, or both.

[0070] That is, users can make modification(s) (to the natural language outline 110, code segment 112 or both) and the system 100 can use the generative neural network 106 to appropriately propagate the modification(s) (by updating the code segment 112, the natural language outline 110, or both, as needed). With this process, users can specify the main idea of each desired change in the modified natural language outline 110, and the system can use the generative neural network 106 to propagate these changes. But also, users can modify the code segment 112 to specify more exact changes. Together, these processes allow users to seamlessly switch between code and natural language as needed, increasing code development efficiency and allowing users to think at a higher level of abstraction more often. Further details of generating updated natural language outline, updated code segment or both will be described below.

[0071] FIG. 2 is a flow diagram of an example process 200 for generating an output sequence defining a natural language outline of the computer code segment. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a natural language generation system, e.g., the natural language generation system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200. The system obtains a code input characterizing a computer code segment that includes a plurality of lines of computer code (step 202).

[0072] As described above, the computer code segment can be any appropriate portion of computer code, e.g., any appropriate portion of a source code file (e.g., a class or a function definition) written in any appropriate programming language (e.g., Python or C++).

[0073] In some cases, the code input includes the computer code segment. For example, if the code segment is a function definition, then the code input can include the function signature (i.e., function name, parameter list, return type) and function body (i.e., lines of computer code to be executed when the function is called). As a particular example, if the code segment is a Python function definition "def sq(x):\n return x**2”, then the code input includes the Python function definition "def sq(x):\n return x**2”.

[0074] In some cases, the code input includes a natural language description of an intended function of the computer code segment. For example, the natural language description can be for definitions (e.g., classes, functions, constants, data structures, data types, packages, aliases, macros, and so on). As a particular example, if the computer code segment intended function is to define a function that receives numeric input, squares the input, and returns the squared input, then the natural language description can be “write a function that receives an input of float type as variable x‘ and returns ‘x‘ multiplied by itself.”

[0075] In some cases, the input sequence includes a respective identifier for each of the plurality of lines of code. For example, the system can prepend each line of the original computer code with its line number. As a particular example, if the original code segment is the two line Python function definition “def sq(x):\n return x**2” then the input sequence can include respective identifiers for each of the plurality as lines such that the input sequence includes “11 def sq(x):\n 2|retum x**2”, where “1|” and “2|” are identifiers for the first line and second line respectively.

[0076] In some cases, the code input includes a prompt that causes the generative neural network to generate the output sequence that defines the natural language outline. For example, the prompt can include instructions and examples of how to generate the output sequence. More generally, in some cases, the prompt incorporates one or more prompting techniques (e.g., zero-shot prompting, few-shot prompting, chain-of-thought prompting, role prompting, instruction prompting, rewriting or refining prompts, output constraints, self-consistency prompting, tool-use prompting, contextual priming, and so on). The system generates, from the code input, an input sequence to a generative neural network (step 204).

[0077] As described above, the system can generate the input sequence by tokenizing the code input. That is, the system can use the code input to generate an input sequence by converting the code input into a sequence of tokens and then converting the sequence of tokens into a sequence of encoded representations (i.e., a sequence of dense embeddings) suitable for processing by a generative neural network (described in further detail below).

[0078] For example, for a code input represented as natural language text, the system can map each character, word, or sub-word of the natural language text to a corresponding token by applying a text tokenizer to the input text. For example, the system can apply the Byte-Pair Encoding (BPE), WordPiece, or SentencePiece tokenizers to divide the natural language text data into tokens from a vocabulary. The system can then process the token sequence with a text embedding layer of the generative neural network that maps the tokens to embeddings to generate a sequence of encoded representations (i.e., embeddings) that the generative neural network can further process.

[0079] In some implementations, when the code input includes the computer code segment, the input sequence includes the lines of computer code from the computer code segment. That is, when the system generates the input sequence by tokenizing the code input represented as natural language text that includes lines of computer code, the system tokenizes the lines of computer code (which include literals, operators, identifiers, and so on) and other artifacts (e g., instructions, examples, and so on) by applying a text tokenizer followed by an embedding layer of the generative neural network).

[0080] The system processes the input sequence using the generative neural network to generate an output sequence defining a natural language outline of the computer code segment (step 206). The natural language outline partitions the computer code segment into logical sections and includes a respective natural language statement for each logical section that summarizes the logical section in natural language.

[0081] The generative neural network can have any of a variety of neural network architectures. That is, the generative neural network can have any appropriate architecture in any appropriate configuration that can process an input sequence to generate an output sequence, including fully connected layers, convolutional layers, recurrent layers, attention-based layers, and so on, as is appropriate.

[0082] In some cases, the generative neural network is a pre-trained neural network (i.e.. the system or another system has previously determined the values of the trainable parameters of the neural network through training on large data sets for one or more general tasks, e.g., next token prediction, image captioning, text-image alignment, and so on).

[0083] In some cases, the generative neural network processes a sequence of tokens to generate, as output, a sequence of tokens from a vocabulary, and the tokens can represent any modality of data such as text, image, audio, video and so on. For example, the generative neural network can be one that belongs to the Gemini family of neural networks, the Gemma family of neural networks, and so on.

[0084] In some implementations, the generative neural network is configured to process a sequence of tokens to auto-regressively generate an output sequence of tokens. That is. in some implementations, the generative neural network can be referred to as an autoregressive neural network when the generative neural network auto-regressively generates an output sequence of tokens. More specifically, the auto-regressively generated output is created by generating each particular token in the output sequence conditioned on a current input sequence that includes any tokens that precede the particular token in the output sequence, e.g., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token.

[0085] For example, the generative neural network can be an auto-regressive Transformer-based neural network that includes (i) a plurality of attention blocks that each apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution.

[0086] In this example, the generative neural network can have any of a variety of Transformer-based neural network architectures. Examples of such architectures include those described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute- optimal large language models, arXiv preprint arXiv:2203.15556, 2022; J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann. H. F. Song, J. Aslanides, S. Henderson, R. Ring. S. Young, E. Rutherford, T. Hennigan, J. Menick. A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Eisen, S. M. Jayakumar. E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini. L. Sifre. L. Martens, X. L. Li. A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin. A. Clark, D. de Las Casas. A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S.

[0087] Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR. abs / 2112. 11446, 2021; Cohn Raffel, Noam Shazeer, Adam Roberts, Kathenne Lee, Sharan Narang. Michael Matena. Yanqi Zhou, Wei Li. and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade. Yifeng Lu. and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.

[0088] Generally, to apply the self-attention operation, each attention block uses one or more attention heads. Each attention head generates a set of queries, a set of keys, and a set of values, and then applies any of a variety7of variants of query7-key-value (QKV) attention, e g., a dot product attention function or a scaled dot product attention function, using the queries, keys, and values to generate an output. Each query, key, value can be a vector that includes one or more vector elements. When there are multiple attention heads, the attention block then combines the outputs of the multiple attention heads, e.g., by concatenating the outputs and, optionally, processing the concatenated outputs through a linear layer.

[0089] Further in some cases, the generative neural network is a fine-tuned neural network (i.e., the system or another system updates previously determined pre-trained values of the trainable parameters of the generative neural network through further training on a task specific data set). During fine-tuning, the generative neural network's parameters (or a subset of them) are further adjusted, adapting the general knowledge acquired during pre-training to the specific nuances of processing an input sequence generate from a code input to generate an output sequence defining a natural language outline for the code input. This approach can improve the performance of generating natural language outlines while requiring less data and computational cost than training the generative neural network from scratch. As described above, in some implementations, the generative neural network processes an input sequence that includes a prompt (e g., zero-shot prompt, few-shot prompt, chain-of-thought prompt, and so on), which can remove the need for fine-tuning when the generative neural network is a pre-trained neural network. That is, the generative neural network can produce high-quality natural language outlines for complex code not seen during training with the inclusion of a prompt in the input sequence, all without requiring any new training or fine-tuning.

[0090] In some cases, when the code input includes the computer code segment, the output sequence includes the respective natural language statements for each logical section interleaved within the computer code segment. That is, the output sequence includes the computer code segment interspersed with the respective natural language statements for each logical section.

[0091] In some cases, within the output sequence, positions of the respective natural language statements for each logical section indicate partitions between the logical sections. That is, the positions of the respective natural language statements for each logical section serve as demarcations that define the disjoint subsets of the lines of code within the code segment included in each logical section.

[0092] In some cases, within the output sequence, each respective natural language statement is designated by one or more pre-determined code symbols. That is, one or more pre-determined code symbols in the output sequence specify portions of the output sequence that correspond to the respective natural language statements.

[0093] For example, the code symbols “<NL statements can be start tag that indicates the start of a natural language statement for a logical section, and “< / NL statements can be the end tag that indicates the end of the natural language statement for the logical section, so that an output sequence that includes “<NL statement Compute all pairwise distances between nodes < / NL statements has a natural language statement '‘Compute all pairwise distances between nodes” between the start tag and end tag.

[0094] In some cases, the one or more pre-determined code symbols indicate a beginning of a code comment. For example, for the Python programming language, the code symbols “#*” can indicate the beginning of a respective natural language statement, where the code symbol indicates a beginning of a code comment. As another example, for the C++ programming language, the code symbols “ / / *” indicate the beginning of a respective natural language statement, where the code symbols “ / / ” indicate the beginning of a code comment. In some cases, the one or more pre-determined code symbols indicate a level of trustworthiness of the respective natural language statement for each of the logical sections. For example, while “#*” can indicate the beginning of a respective natural language statement, “#*!” can indicate the respective natural language statement is verified as being correct. A natural language statement is “verified’' means that the system, a user, or another system has determined that the statement accurately summarizes the logical section it refers to.

[0095] FIG. 3 is an example 300 of an output sequence depicted as a right-to-left, line-by- line text defining a natural language outline (i.e., the natural language outline 110).

[0096] In particular, example 300 includes four logical sections 302A-D where the respective natural language statements for each logical section is interleaved within the computer code segment. The positions of each natural language statement indicate partitions between logical sections, and each natural language statement is designated by code symbols where the code symbol indicates the beginning of a code comment.

[0097] In some implementations, as part of step 206, when the system processes the input sequence that includes the code segment using the generative neural network, the system performs constrained generation to cause the generative neural network to copy lines of code from the input sequence to the output sequence. That is, the system uses a constrained generation approach to ensure faithful code reproduction of the code segment in the output sequence by utilizing constrained decoding, by applying a strict, regex-like template that compels the generative neural network to replicate the code segment verbatim, while strategically granting it the flexibility to interleave the original lines of code with natural language statements.

[0098] For example, the system can obtain defining constraints (i.e., constraints for the output sequence, e.g., in terms of regular expressions, or context-free grammars), and, for each output token of the output sequence the system generates using the generative neural network, the system can modify the probability of selecting a particular output token over all possible output tokens in a vocabulary in accordance with the defining constraints.

[0099] So, as a particular example, the system can receive a defining constraint (e.g., from a user or another system) that causes the system to apply a mask to restrict output sequences to adhere to the defining constraints (i.e., outputting exact copies of lines of code in the input sequence). That is, when the system selects a token representing the first token of a sequence of tokens that represents a line of code in the input sequence, the system subsequently will select the remaining sequence of tokens representing the line of code through the appropriate application of masks at each output position of the output sequence until the line of code is reproduced in the output sequence. In other words, when selecting output tokens that correspond to a line of code in the input sequence, the system masks out (i.e., sets the probability of selection to zero) any token that is not a valid next token. But, when the system is selecting an output token that is not part of a line of code, the system does not use a mask to modify the probability over possible output tokens. In this way, the system can generate an output sequence that includes the exact lines of code present in the input sequence and includes natural language statements anywhere between these lines of code.

[0100] As an illustrative example, for the example output sequence of FIG. 3, any line of text that does not begin with “#*” (i.e., code symbols that designate a natural language segment) are lines of code of the code segment from the input sequence and can have been generated using constrained generation, e.g., as described above, to exactly match the input sequence. But lines of text that begin with “#*” are those the system generates with unconstrained generation.

[0101] In some implementations, when the input sequence includes a respective identifier for each of the plurality7of lines of code, the output sequence includes, for each natural language statement, a respective identifier of a line of code from the plurality of lines of code to which the natural language statement corresponds.

[0102] For example, the code input can include code line numbers as respective identifiers for each of the plurality of lines of code; and the code input can also include a prompt that is a natural language instruction (i.e., a zero-shot prompt) to output a sequence of natural language statements along with respective code line numbers of where the statements should be added in the code segment and the text of the statements themselves.

[0103] As an example, if the input sequence is " I |def sq(x):\n 2|retum x**2”, where the “1|” and “2|” are code line identifiers for line 1 and line 2 respectively, the system can generate an output sequence that includes “2|Squares the input”, e.g.. by processing a prompt with appropriate instructions as described above, where “2|” in the output sequence indicates that the natural language statement “Square the input” corresponds to code line 2.

[0104] In some implementations, after step 206. the system provides the natural language outline for presentation to a user on a user device. The user device can be any of a variety of appropriate user devices that the system can use to present the natural language outline. For example, the system can provide the outline for presentation to a user of a smart phone, tablet, or laptop through the user device display (i.e., the touch screen for the smart phone and tablet and the display screen for the laptop).

[0105] In some cases, as part of the system providing the natural language outline for presentation to a user on a user device, the system provides the natural language outline for presentation in an integrated development environment (IDE) on the user device. Generally, an IDE is a graphical user interface (GUI) for a software application that provides tools for programmers to develop computer code. So, for example, an IDE can be software running on a laptop user device that displays the natural language outline within its graphical user interface to the user via the laptop display screen.

[0106] In some cases, as part of the system providing the natural language outline for presentation to a user on a user device, the system provides the natural language statements for presentation interleaved with the lines of code (i.e., the lines of computer code of the computer code segment). That is, the natural language statements are presented alongside the lines of code.

[0107] In other cases, as part of the system providing the natural language outline for presentation to a user on a user device, the system provides the natural language statements for presentation without the lines of code (i.e., the lines of computer code of the computer code segment). That is, the natural language statements are presented but not alongside the lines of code.

[0108] In some implementations, as part of the system providing the natural language outline for presentation to a user on a user device, the system provides the natural language statements for presentation separate from the lines of code in a user interface. That is, each natural language statement is presented in the user interface in association with a respective user interface element that when selected by the user causes the corresponding logical section of computed code to be presented in the user interface.

[0109] For example, the system can present the natural language statements as "‘accordion panel” headers within a GUI of an IDE displayed on a laptop such that if a user were to click a natural language statement, then the header expands to reveal contents that include the corresponding logical section of computer code.

[0110] FIG. 4 shows an example 400 of a natural language outline (i.e., the natural language outline 110) for presentation to a user on a user device. In particular, example 400 presents the natural language outline 110 in an integrated development environment (IDE) via a GUI on a laptop user device with the natural language statements interleaved with the lines of code of the computer code segment in the right display pane and with the natural language statements displayed without the lines of code in the left display pane.

[0111] The left display plane of the IDE shows that respective natural language outlines for different functions. The natural language outline 402 is for the function named “nearest_neighbor_tour” in the right display plane. Clicking a particular natural language statement in the natural language outline displayed in the left pane can move the cursor to the corresponding code location in the right pane. For example, clicking the natural language statement ‘Iteratively add the nearest unvisited node to the tour' in the left display pane brings the cursor in the right pane to the corresponding code line 404 (i.e., the 39thcode line) as indicated by the page wide horizontal lines.

[0112] As described above, in the right display pane is the natural language outline 110 shown interleaved with the code segment, either as non-code annotations or as code comments. For example, the natural language statement 408 “Compute the distance matrix between all nodes” summarizes the logical section that includes the portion of computer code segment “distances = scipy. spatial. distance_matrix(nodes, nodes)”.

[0113] The natural language statement 406 “Initialize the tour starting from the first node” is an example of a natural language statement presented separate from the lines of code in a user interface with a respective user interface element (i.e., the clickable text) that when selected by the user causes the corresponding logical section of computer code to be presented in the user interface. In FIG. 4 the natural language statement 406 has not been selected yet (as indicated by the right pointing arrow) which is why the logical section is not shown. But if the natural language statement 406 were to be selected, the right pointing arrow would become a downwards pointing arrow and the corresponding logical section would be presented. So, natural language outlines can offer “intuitive code folding”. The other natural language statements in the right pane display their corresponding logical section (as is indicated by the downwards pointing arrows preceding each natural language statement).

[0114] In some implementations, after the system generates an output sequence defining a natural language outline, the system generates, from the natural language outline, another input sequence and processes the other input sequence using the generative neural network to generate the computer code segment. In other words, the system can process an input sequence generated from a code input using a generative neural network to generate an output sequence defining a natural language outline of the computer code segment. The system can then process another input sequence generated from the natural language outline using the generative neural network to generate the computer code segment.

[0115] The input sequence to generate the computer code segment can represent the same types of data as the input sequence to generate the natural language outline described above can represent (e.g., can include a prompt that guides the generation of an output sequence and can be a sequence of tokens representing text, code, images, audio, video). Also, the system can process the input sequence to generate the computer code segment using the generative neural network using the same methods described above that the system uses the generative neural network to process the input sequence to generate the natural language outline (e.g., auto-regressively processing a sequence of tokens to generate an output sequence).

[0116] As a particular example, a code input characterizing a computer code segment can include a natural language description of the code segment (but no computer code lines). For example, the code input could include the natural language text “a function that uses a ‘greedy’ strategy' for finding a short route that visits a series of locations, often called the ‘Nearest Neighbor’ algorithm.” and the resulting natural language outline can be. e.g., Table 1, where each bullet point indicates a natural language statement. _

[0117] Table 1

[0118] The system can then generate from the natural language outline (i.e., Table 1) an input sequence to generate the computer code segment (i.e., a tokenization of the natural language outline) which includes a prompt that includes instructions to generate the corresponding code segment to the natural language outline. The system can then process the input sequence to generate the computer code segment using the generative neural network to generate an output sequence that represents the computer code segment. For example, the resulting computer code segment can be that represented by the output sequence 110 shown in FIG. 3. Using an input sequence to generate the computer code segment that includes the natural language outline, as described above, results in the code segment more reliably reflecting the intended structure expressed in the natural language outline, and therefore improves the performance of the resulting code segment. That is, using the generative neural network to process the input sequence to generate the computer code segment which includes the natural language statements for each logical section included in the natural language outline makes the generation of the code segment more manageable because each logical section of the code will be based on a corresponding natural language statement. Also, by basing the generation of each logical section of the code segment on the respective natural language statement, the code segment will accurately reflect the intended processes represented by the natural language statements (and by extension the natural language outline). Without this natural language outline, the generation of the code segment will be less likely to adhere to an intended structure, which leads to higher variability of generated code segment that may not function as intended.

[0119] In some cases, the code input includes lines of computer code characterizing a computer code segment, but the computer code segment the system generates by processing the input sequence to generate the computer code segment that includes the natural language outline includes different lines of computer code than that included in the code input. That is. the system can generate the natural language outline from a code input that includes a code segment that is not necessarily the same code segment generated by the system when it processes the input sequence to generate the computer code segment that includes the natural language outline using the generative neural network. Such a code segment generated by the system processing the input sequence to generate the computer code segment that includes the natural language outline may be an improved code segment. For example, the original code segment of the code input and the newly generated code segment may both be accurately described by the same natural language outline, but the second input sequence that includes the natural language outline may also include instructions to generate a code segment based on the natural language outline such that the newly generated code segment is memory efficient (i.e., has as small a memory' footprint as possible) during run time of the code segment on hardware. As a result, the new code segment may be more memory efficient and therefore improved and different from the original code segment. In some implementations, as part of generating the input sequence to generate the computer code segment from the natural language outline, the system provides the natural language statements for presentation to the user. The system then receives a user input specifying one or more modifications to the presented natural language statements to generate a new plurality7of natural language statements. Then, the system generates an input sequence to generate the computer code segment that includes the new plurality7of natural language statements.

[0120] By having the generative neural network generate the computer code segment aligned with a user modified natural language outline (i.e., user modified natural language statements), the system will generate a code segment that satisfies the user and is easier for the user to interpret because they already expect the outlined approach in the code segment.

[0121] In some implementations, after the system generates an output sequence defining a natural language outline, the system receives one or more user inputs modifying the natural language outline to generate a modified natural language outline. The system then generates, from the modified natural language outline, another input sequence. Then, the system processes the other input sequence using the generative neural network to generate an updated code segment. That is, the system can process a modified natural language outline using an input sequence to generate an updated code segment that reflects the modified natural language outline. When the system generates the updated code segment it necessarily follows that the code segment previously existed.

[0122] The one or more user inputs modifying the natural language outline can be additions, deletions, or alterations to the natural language outline. For example, when the system presents the natural language outline in an IDE on a laptop user device display screen, the one or more user inputs can be user laptop keyboard inputs that correspond to additions, deletions, or alterations of the presented natural language outline.

[0123] The input sequence to generate an updated code segment can represent the same types of data as the input sequence to generate a natural language outline described above can represent (e.g.. can include a prompt the guides the generation of an output sequence and can be a sequence of tokens representing text, code, images, audio, video). Also, the system can process the input sequence to generate an updated code segment using the generative neural network using the same methods described above that the system uses the generative neural network to process the input sequence to generate the natural language outline (e.g., auto-regressively processing a sequence of tokens to generate an output sequence).

[0124] In some implementations, the input sequence to generate an updated code segment includes the code segment and the modified natural language outline. For example, the system tokenizes both the code segment and the natural language outline to generate the input sequence to generate an updated code segment.

[0125] In some implementations, the one or more user inputs further modify the code segment to generate an initial updated code segment, and the input sequence to generate an updated code segment includes the modified natural language outline and the initial updated code segment.

[0126] For example, when the system presents the natural language outline and code segment in an IDE on a laptop user device display screen, the one or more user inputs can be user laptop keyboard inputs that correspond to additions, deletions, or alterations of the presented natural language outline and the code segment. The system then tokenizes the modified natural language outline and code segment to generate the input sequence to generate an updated code segment.

[0127] In some implementations, the system begins the process of generating the input sequence to generate an updated code segment upon system determined conditions being satisfied. For example, at pre-defined time intervals (e.g., 0.1 second, 1 second, or 10 second intervals), whenever there are no user inputs being received, the system checks for previous one or more user inputs that modify the natural language outline or the code segment and, if the system detects modifications, the system begins generating the input sequence to generate an updated code segment.

[0128] In some implementations, the system begins the process of generating the input sequence to generate an updated code segment upon receiving a user input to do so. That is, after the system receives one or more user inputs that modify the natural language outline or the code segment, the system can receive a user input that explicitly initiates the process of generating the input sequence to generate an updated code segment.

[0129] FIG. 5 is an example 500 of a modified natural language outline and an updated code segment presented in an IDE on a user device such as a laptop or desktop computer display.

[0130] In particular, the first display pane 502 shows the natural language outline with natural language statements interleaved within the code segment. But the first displaypane 502 also shows the result of a user input that modifies a natural language statement on line 17 to be "#* Open the CSV file and read each row” from an original natural language statement of “#* Open the JSON file and read each row”. The code segment in display pane 502 reflects the original natural language statement, e.g., as can be seen on line 19 with the Python code to load a JSON file (i.e., “json.load(f)”).

[0131] The second display pane 504 shows the updated code segment that reflects the modified natural language outline as a diff (i.e., a diff is a comparison of previous source code file and updated source code file to highlight their differences). The lines prefixed with the symbol represent previous source code file lines and the lines prefixed with the symbol “+” represent updated source code file lines. The updates to the code segment include an updated docstring and replacement of JSON logic with analogous CSV logic, including adding string-to-number conversions which were not needed before. As a particular example, the previous Python code at line 19 '‘for row in json.load(f)” is updated to be “reader = csv.DictReader(f)\n for row in reader:” to reflect the fact that the natural language statement has been modified to open a CSV file instead of a JSON file.

[0132] Example 500 illustrates the advantages of the described techniques.

[0133] For example, example 500 shows that concise modifications to the natural language outline results in extensive updates throughout the code segment in the updated code segment when the system processes an input sequence to generate an updated code segment that reflects the modified natural language outline. As illustrated in example 500, modifying a single word in line 17 (i.e., replacing “CSV” with “JSON” as described above) results in multiple updated code lines in the updated code segment (e.g., docstring change and code logic). Therefore, the natural language outline allows for the efficient and accurate generation of an updated code segment. Without the natural language outline to modify, generating an updated code segment would require the use of extensive, cumbersome inputs to detail desired changes of a user.

[0134] In some implementations, after the system generates an output sequence defining a natural language outline, the system receives one or more user inputs modifying the computer code segment to generate a modified computer code segment. The system then generates, from the modified computer code segment, another input sequence. Then, the system processes the other input sequence using the generative neural network to generate an updated natural language outline for the modified computer code segment.

[0135] The one or more user inputs modifying the computer code segment can be additions, deletions, or alterations to the computer code segment. For example, when the system presents the natural language outline in an IDE on a laptop user device display screen, the one or more user inputs can be user laptop keyboard inputs that correspond to additions, deletions, or alterations of the presented computer code segment.

[0136] The input sequence to generate an updated natural language outline can represent the same types of data as the input sequence to generate a natural language outline described above can represent (e.g., can include a prompt the guides the generation of an output sequence and can be a sequence of tokens representing text, code, images, audio, video). Also, the system can process the input sequence to generate an updated natural language outline using the generative neural network using the same methods described above that the system uses the generative neural network to process the input sequence to generate the natural language outline (e.g., auto-regressively processing a sequence of tokens to generate an output sequence).

[0137] In some implementations, the input sequence to generate an updated natural language outline includes the modified code segment, and the natural language outline. For example, the system tokenizes both the modified code segment and the natural language outline to generate the input sequence to generate an updated natural language outline.

[0138] In some implementations, the one or more user inputs further modify the natural language outline to generate an initial updated natural language outline, and the input sequence to generate an updated natural language outline includes the initial updated natural language outline and the modified code segment.

[0139] For example, when the system presents the natural language outline and code segment in an IDE on a laptop user device display screen, the one or more user inputs can be user laptop keyboard inputs that correspond to additions, deletions, or alterations of the presented natural language outline and the code segment. The system then tokenizes the modified natural language outline and code segment to generate the input sequence to generate an updated natural language outline.

[0140] In some implementations, the system begins the process of generating the input sequence to generate an updated natural language outline upon system determined conditions. For example, at pre-defined time intervals (e.g.. 0.1 second. 1 second, or 10 second time intervals), whenever there are no user inputs being received, the system checks for previous one or more user inputs that modify the natural language outline or the code segment and, if modifications are detected, the system begins generating the input sequence to generate an updated natural language outline. In some implementations, the system begins the process of generating the input sequence to generate an updated natural language outline upon receiving a user input to do so. That is, after one or more user inputs that modify the natural language outline or the code segment, the system can receive a user input that explicitly initiates the process of generating the input sequence to generate an updated natural language outline.

[0141] FIG. 6 is an example 600 of a modified code segment and an updated natural language outline presented in an IDE on a user device such as a laptop or desktop computer display.

[0142] In particular, the first display pane 602 shows the natural language outline with natural language statements interleaved within the code segment. But the first display pane 602 also shows the result of a user input that modifies the code segment, i.e., the user modifies line 27 to include “* p[“price”]” which is absent in the original code segment. The natural language statement above the modified line 27 “#* Calculate the total quantity of all products.” in display pane 602 reflects the original code segment.

[0143] The second display pane 604 shows the updated natural language outline that reflects the modified code segment as a diff (i.e., a diff is a comparison of previous source code file and updated source code file to highlight their differences). The lines prefixed with the symbol represent previous source code file lines and the lines prefixed with the symbol “+” represent updated source code file lines. The updates to the natural language outline include an updated natural language statement for line 27 “#* Calculate the total value of all products” where the word “value” has replaced the word “quantify” and correctly reflects the modified code segment.

[0144] Example 600 illustrates the advantages of the described techniques.

[0145] For example, example 600 shows that modifications to the code segment results in updates to the natural language outline, and the updates to the natural language outline can help validate the code modifications. As illustrated in example 600 and described above, modifying line 27 to include “*p[“price”]” changes the meaning of the “total” being calculated from “quantity” to “value”. The updated natural language statements (i.e., code lines 26 and 28, and 30) of the updated natural language outline allows the system to validate this modification by verifying the natural language statements are consistent with each other and the modified code (i.e., the system, a user, or another system can verify that these updated natural language statements make sense). Without the natural language outline to update, a modification to the code segment could not be so easily validated. In some implementations, the operations described above can be performed as part of a larger technique. That is, the described techniques can be incorporated into various applications.

[0146] An example application is software security assessment. That is, the described techniques can be a part of, e.g., mobile application (app) security assessment to determine whether the mobile app is deceptive or malicious before publishing the mobile app onto an online app store.

[0147] As a particular example, software security assessment for the evaluation of a source code file may include using a generative neural network to generate three kinds of predictions per function in the source code file, designed for different benefits: (1) a suspicion score that is an integer from 0 (not suspicious at all) to 3 (very suspicious) which is extremely quick to read and allows more nuance than a simple binary prediction and enables programmatic processing due to its simplicity7; (2) a paragraph-form summary, e.g., of about 1-3 sentences, describing what the function does and why it is suspicious or not, in a format that is a baseline for generative neural network-based code explanation and is familiar to users; and (3) lastly, a natural language outline interleaved in the code segment in the source code file. The natural language outline allows a user to quickly identify problematic areas of code by reading the natural language statements before reading the code segment for faster understanding and localization of the malware implementation, without non-security-related distractions.

[0148] FIG. 7 is an example 700 of an application that includes the use of the described techniques. In particular, example 700 shows the above described suspicion score, summary, and natural language outline for a function included in a source file for a mobile app. The natural language statements of the natural language outline are interleaved in the code segment and make it very easy to identify problematic lines of code (e.g., the natural language statements “ / / Accesses the user’s clipboard.” and ‘7 / Sends the clipboard contents to ‘this.abl.c23()’”) reveal obvious security threat that would not be as readily apparent reading the code segment.

[0149] FIG. 8 is an example 800 of the performance of the described techniques.

[0150] In particular, example 800 shows a table that summarizes the rate of “issues” (i.e., none, major, or minor) that arise when generating a natural language outline for each of 30 Python functions using five different generative neural networks (i.e., Gemini 1.5 Pro, Gemini 1.5 Flash, etc.) and output sequences that include the respective natural language statements interleaved within the computer code segment (i.e., “Interleaved Generation” in the techniques column of the table) or output sequences that include, for each natural language statement, a respective identifier of a line of code from the plurality of lines of code to which the natural language statement corresponds (i.e., '‘Line Number Infilling” in the techniques column of the table).

[0151] Some examples of the major issues refer to the output sequence defining the natural language outline reflecting: 1) modified (e.g., removed or added) code lines for the code segment relative to the code segment present in the input sequence; and 2) no natural language statements included in the natural language outline.

[0152] Some examples of the minor issues refer to the output sequence defining the natural language outline reflecting: 1) addition or removal of blank lines to the code lines for the code segment relative to the code segment present in the input sequence; and 2) removed code comment line for the code segment relative to the code segment present in the input sequence.

[0153] No issues refer to the absence of any of the above described major or minor issues.

[0154] The table of example 800 shows that the described techniques consistently produce accurate natural language outlines. In particular, the use of a generative neural network that is Gemini 1.5 (Pro or Flash) produces zero major errors.

[0155] The table of example 800 also shows the average number of natural language statements per natural language outline. Too many natural language statements often mean poor abstraction of ideas, but too few may indicate a lack of detail.

[0156] The table of example 800 shows that the described techniques consistently produce minimal number of natural language statements per natural language outline, indicating the presence of concise and informative natural language statements.

[0157] This specification uses the term ‘'configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0158] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0159] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be. or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0160] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network. In this specification, the term “database’' is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[0161] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0162] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g.. an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0163] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory’ can be supplemented by, or incorporated in, special purpose logic circuitry’. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g.. a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e g., a universal serial bus (USB) flash drive, to name just a few.

[0164] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0165] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user bysending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0166] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0167] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework.

[0168] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet. The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0169] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0170] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0171] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0172] What is claimed is:

Claims

CLAIMS1. A method performed by one or more computers, the method comprising: obtaining a code input characterizing a computer code segment comprising a plurality of lines of computer code; generating, from the code input, an input sequence to a generative neural network; and processing the input sequence using the generative neural network to generate an output sequence defining a natural language outline of the computer code segment, wherein the natural language outline partitions the computer code segment into logical sections and comprises a respective natural language statement for each logical section that summarizes the logical section in natural language.

2. The method of claim 1, wherein the code input comprises the computer code segment.

3. The method of claim 2. wherein the input sequence comprises the lines of computer code from the computer code segment.

4. The method of claim 2 or 3, wherein the output sequence comprises the respective natural language statements interleaved within the computer code segment.

5. The method of claim 4, wherein, within the output sequence, positions of the respective natural language statements indicate partitions between the logical sections.

6. The method of claim 4 or claim 5, wherein, within the output sequence, each respective natural language statement is designated by one or more pre-determined code symbols.

7. The method of claim 6, wherein the one or more pre-determined code symbols indicate a beginning of a code comment.

8. The method of any one of claims 4-7, wherein processing the input sequence using the generative neural network comprises performing constrained generation tocause the generative neural network to copy lines of code from the input sequence to the output sequence.

9. The method of any one of claims 3-8, wherein the input sequence comprises a respective identifier for each of the plurality of lines of code.

10. The method of claim 9. wherein the output sequence comprises, for each natural language statement, a respective identifier of a line of code from the plurality of lines of code to which the natural language statement corresponds.

11. The method of any preceding claim, wherein the code input comprises a natural language description of an intended function of the computer code segment.

12. The method of any preceding claim, wherein the code input comprises a prompt that causes the generative neural network to generate the output sequence that defines the natural language outline.

13. The method of any preceding claim, wherein the generative neural network is configured to process a sequence of tokens to auto-regressively generate an output sequence of tokens.

14. The method of any preceding claim, further comprising: providing the natural language outline for presentation to a user on a user device.

15. The method of claim 14, wherein the providing the natural language outline for presentation to a user on a user device comprises providing the natural language outline for presentation in an integrated development environment (IDE) on the user device.

16. The method of claim 14 or claim 15. wherein the providing the natural language outline for presentation to a user on a user device comprises providing the natural language statements for presentation interleaved with the lines of code.

17. The method of claim 14 or claim 15. wherein the providing the natural language outline for presentation to a user on a user device comprises providing the naturallanguage statements for presentation without the lines of code.

18. The method of claim 14 or claim 15, wherein the providing the natural language outline for presentation to a user on a user device comprises providing the natural language statements for presentation separate from the lines of code in a user interface, wherein each natural language statement is presented in the user interface in association with a respective user interface element that when selected by the user causes the corresponding logical section of computed code to be presented in the user interface.

19. The method of any preceding claim, further comprising: generating, from the natural language outline, a second input sequence; and processing the second input sequence using the generative neural network to generate the computer code segment.

20. The method of any preceding claim, wherein generating, from the natural language outline, a second input sequence comprises: providing the natural language statements for presentation to the user; receiving a user input specifying one or more modifications to the presented natural language statements to generate a new plurality of natural language statements; and generating a second input sequence that comprises the new plurality' of natural language statements.

21. The method of any preceding claim, further comprising: receiving one or more user inputs modifying the natural language outline to generate a modified natural language outline; generating, from the modified natural language outline, a third input sequence; and processing the third input sequence using the generative neural network to generate an updated code segment.

22. The method of claim 21, wherein the third input sequence includes the code segment, and the modified natural language outline.

23. The method of claim 21, wherein the one or more user inputs further modify the code segment to generate an initial updated code segment, and wherein the third input sequence includes the modified natural language outline and the initial updated code segment.

24. The method of any preceding claim, further comprising: receiving one or more user inputs modifying the computer code segment to generate a modified computer code segment; generating, from the modified computer code segment, a fourth input sequence; and processing the fourth input sequence using the generative neural network to generate an updated natural language outline for the modified computer code segment.

25. The method of claim 24, wherein the fourth input sequence includes the modified code segment, and the natural language outline.

26. The method of claim 24, wherein the one or more user inputs further modify the natural language outline to generate an initial updated natural language outline, and wherein the fourth input sequence includes the initial updated natural language outline and the modified code segment.

27. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-26.

28. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-26.

Images