System and method for an conversational framework for program synthesis

Abstract

This system and method provides an interactive framework for program synthesis. [Solution] The embodiments described herein provide a program synthesis framework that generates a code program through a multi-turn conversation between a user and a system. Specifically, a description for solving a target problem is broken down into several steps, each step including a description in natural language (prompt) that is input to a generation model as a user utterance. The model then synthesizes a functionally correct subprogram according to the current user utterance and taking into account the description and synthesized subprograms from the previous step. The subprograms generated in the steps are then combined to form the output of a program that responds to the target problem.

Description

[Technical Field]

[0001] This disclosure is a non-provisional application, U.S. Provisional Application No. 63 / 317,244, filed on 7 March 2022, claiming priority to U.S. Patent Application No. 17 / 889,998, filed on 17 August 2022, which is expressly incorporated herein by reference in its entirety.

[0002] [Technical field] This disclosure relates, in general, to machine learning models and automated code generation, and more specifically, to the pre-training of machine learning models for conversational program synthesis. [Background technology]

[0003] Program synthesis is the process of automatically generating programs in a programming language (such as Python®, C++, or Pearl) that satisfy user intents expressed in some form of specification (e.g., natural language), often via machine learning models. Successful program synthesis improves the productivity of experienced programmers and makes programming accessible to a much broader user base. However, long-term challenges in program synthesis remain unresolved. For example, maintaining a rich and expressive search space often requires a large search space of programming code. Therefore, efficiently exploring a large number of candidates in such a vast search space remains challenging.

[0004] As another example, providing a proper specification of user intent for code generation remains challenging. User intent for program synthesis can be a logical specification specifying logical relationships between program inputs and outputs, input-output examples, or natural language. Compiling the specification alone can sometimes require as much effort as the program generation itself, making it impractical for users to provide a complete formal or informal specification. However, simpler specifications, such as those using mere input-output examples or abstract descriptions in natural language, may not adequately specify the problem and could lead to undesirable solutions.

[0005] Therefore, program synthesis techniques are needed to address these problems. [Brief explanation of the drawing]

[0006] [Figure 1] This is a simplified diagram of a computing device for implementing an interactive program synthesis system, according to several embodiments. [Figure 2] Figure 1 is a simplified block diagram of a networked system suitable for implementing the program synthesis framework and other embodiments described herein. [Figure 3] This is an exemplary logic flow diagram illustrating a method for synthesizing code programs for a target problem based on the framework shown in Figures 1 and 2, according to some embodiments described herein. [Figure 4] These are exemplary logic flow diagrams illustrating how to train the program synthesis modules shown in Figures 1-2 in parallel on multiple processors, according to some embodiments described herein. [Figure 5] This is a simplified block diagram illustrating an example of program synthesis via multi-turn conversation according to the embodiments described herein. [Figure 6]These are exemplary data performance charts showing exemplary performance of the conversational program synthesis module described in Figures 1-5, according to embodiments described herein. [Figure 7] These are exemplary data performance charts showing exemplary performance of the conversational program synthesis module described in Figures 1-5, according to embodiments described herein. [Figure 8] These are exemplary data performance charts showing exemplary performance of the conversational program synthesis module described in Figures 1-5, according to embodiments described herein. [Figure 9] These are exemplary data performance charts showing exemplary performance of the conversational program synthesis module described in Figures 1-5, according to embodiments described herein. [Figure 10] These are exemplary data performance charts showing exemplary performance of the conversational program synthesis module described in Figures 1-5, according to embodiments described herein.

[0007] In the diagram, elements with the same name have the same or similar functions. [Modes for carrying out the invention]

[0008] As used herein, the term “Network” may include any hardware or software-based framework, including any artificial intelligence network or system, a neural network or system, and / or any training or learning model implemented on or with it.

[0009] As used herein, the term “module” may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0010] In view of the challenges in efficient searching and user intent specification in program synthesis, the embodiments described herein provide a program synthesis framework that generates a code program through a multi-turn conversation between a user and a system. Specifically, a description for solving a target problem is broken down into several steps, each step including a description in natural language (prompt) that is input to the generation model as a user utterance. The model then synthesizes a functionally correct subprogram according to the current user utterance and taking into account the description and synthesized subprograms from the previous step. The subprograms generated in the steps are then combined to form the output of a program that responds to the target problem.

[0011] In another embodiment, the generated program may also be inaccurate when the code generation model has high uncertainty. The program synthesis framework may generate questions to clarify the user intent. In this way, the user's answers to the system's questions can be used as the user intent.

[0012] In one embodiment, the user intent for program synthesis is used as a specification that specifies logical relationships between program inputs and outputs, input-output examples, or natural language. Since most programmers do not develop a comprehensive vision of their intent until the programming process is well underway, an interactive program synthesis technique may be employed. In this technique, the user interacts with the synthesis system by gradually providing specifications in natural language while receiving responses from the system in the form of synthesized subprograms, thereby enabling the user and the system to complete the program interactively in a multi-turn conversation.

[0013] In one embodiment, the program synthesis framework may be built on top of a Transformer model used to understand the dependencies between sequence elements through an attention mechanism. For example, a large autoregressive transformer is trained on a large Python dataset powered by Cloud Tensor Processing Units (TPUs).

[0014] In one embodiment, due to the large size of the Transformer model for program synthesis, a parallel processing technique is employed that allows the same computation to be performed in parallel on different devices for different input data in order to improve training efficiency. Specifically, for data parallel processing, batches of training data are partitioned and distributed to individual TPU virtual machines (VMs). For model parallel processing, two schemes are supported for sharding model parameters: (1) intra-TPU-VMs where parameters are sharded across matrix unit (MXU) cores in a physical TPU board and replicated across boards, and (2) inter-TPU-VMs where parameters are sharded across TPU-v4 boards and activations are replicated.

[0015] Figure 1 is a simplified diagram of a computing device for implementing an interactive program synthesis system in several embodiments. As shown in Figure 1, the computing device 100 includes a processor 110 coupled to memory 120. The operation of the computing device 100 is controlled by the processor 110. Although the computing device 100 is shown to have one processor 110, it is understood that the processor 110 may represent one or more central processing units, multicore processors, microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), and / or similar within the computing device 100. The computing device 100 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0016] Memory 120 may be used to store software executed by computing device 100 and / or one or more data structures used during the operation of computing device 100. Memory 120 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0017] Processor 110 and / or memory 120 can be arranged in any suitable physical arrangement. In some embodiments, processor 110 and / or memory 120 can be implemented on the same substrate, within the same package (e.g., system in package), on the same chip (e.g., system on chip), and / or the like. In some embodiments, processor 110 and / or memory 120 can include distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, processor 110 and / or memory 120 can be installed within one or more data centers and / or cloud computing facilities.

[0018] In some examples, memory 120 can include a non-transitory tangible machine-readable medium that includes executable code that, when executed by one or more processors (e.g., processor 110), can cause the one or more processors to execute the methods described in more detail herein. For example, as shown, memory 120 can include instructions for program synthesis module 130 that can be used to implement and / or emulate systems and models and / or to implement any of the methods described further herein. In some examples, online compliance module 130 can receive input 140, such as natural language input specifying user intent, via, for example, data interface 115. Program synthesis module 130 can generate output 150 of program snippets in response to input 140.

[0019] For example, the program synthesis module 130 may include a bidirectional t-transformer encoder 131 and an autoregressive transformer decoder 132 for learning normal next token prediction language modeling. Thus, the architecture of the program synthesis module 130 may follow a Transformer decoder with left-to-right causal masking. The encoder 131 may adopt rotary position embedding for positional encoding (for more details on positional encoding, see Su et al., Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021). In the case of the forward pass, the encoder 131 and the decoder 132 may include self-attention circuits and feed-forward circuits in parallel to improve communication overhead. Specifically, the module output may be generated as follows: x t+1 =x t +attn(ln(x t ))+mlp(ln(x t )) Therefore, the calculations of self-attention attn( ) and feed-forward mlp( ) using layer normalization ln( ) can be implemented simultaneously. During implementation, the selection of the architecture and hyperparameters is optimized especially for the hardware layout of TPU-v4.

[0020] In some examples, the program synthesis module 130 may be implemented using hardware, software, and / or a combination of hardware and software.

[0021] Some examples of computing devices, such as computing device 100, may include a non-temporary tangible machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 110), causes one or more processors to execute the process of the Method. Some common forms of machine-readable medium that may contain the process of the Method are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0022] Figure 2 is a simplified block diagram of a networked system suitable for implementing the program synthesis framework described in Figure 1 and other embodiments described herein. In one embodiment, block diagram 200 shows a system including a user device 210 that can be operated by a user 240, data vendor servers 245, 270, and 280, a server 230, and other forms of devices, servers, and / or software components that operate to perform various methods according to the embodiments described. Illustrative devices and servers may include devices, standalones, and enterprise-class servers that run an OS such as MICROSOFT® OS, UNIX® OS, LINUX® OS, or other suitable device and / or server-based OS, similar to the computing device 100 described in Figure 1. It can be understood that the devices and / or servers shown in Figure 2 may be deployed in other ways, and the operations performed and / or services provided by such devices and / or servers may be combined or separated for a given embodiment, and may be performed by a larger or smaller number of devices and / or servers. One or more devices and / or servers may be operated and / or maintained by the same or different entities.

[0023] The user device 210, data vendor servers 245, 270, and 280, and server 230 can communicate with each other via network 260. The user device 210 may be used by a user 240 (e.g., a driver, system administrator, etc.) to access various features available on the user device 210, which may include processes and / or applications associated with server 230 to receive output data anomaly reports.

[0024] The user device 210, the data vendor server 245, and the server 230 may each include one or more processors, memory, and other suitable components for executing instructions such as program code and / or data stored on one or more computer-readable media to implement the various applications, data, and steps described herein. For example, such instructions may be stored on one or more computer-readable media, such as memory or data storage devices, which are internal to and / or external to the various components of the system 200, and / or accessible via the network 260.

[0025] The user device 210 may be implemented as a communication device that can utilize appropriate hardware and software configured for wired and / or wireless communication with the data vendor server 245 and / or server 230. For example, in one embodiment, the user device 210 may be implemented as an autonomous vehicle, a personal computer (PC), a smartphone, a laptop / tablet computer, a wristwatch with appropriate computer hardware resources, eyeglasses with appropriate computer hardware (e.g., GOOGLE GLASS®), other types of wearable computing devices, an embedded communication device, and / or other types of computing devices capable of transmitting and / or receiving data, such as an APPLE® or iPad®. Although only one communication device is shown, multiple communication devices may function similarly.

[0026] The user device 210 in Figure 2 includes a user interface (UI) application 212 and / or other applications 216, which may correspond to executable processes, procedures, and / or applications with associated hardware. For example, the user device 210 may receive a message from the server 230 indicating a synthesized code program and display that message via the UI application 212. In other embodiments, the user device 210 may include additional or different modules having dedicated hardware and / or software, as needed.

[0027] In various embodiments, the user device 210 may include other applications that may be desired in certain embodiments to provide functionality to the user device 210. For example, other applications 216 may include security applications for implementing client-side security features, programmatic client applications for interfacing with appropriate application programming interfaces (APIs) over the network 260, or other types of applications. Other applications 216 may also include communication applications such as email, texting, voice, social networking, and IM applications that enable the user to send and receive emails, calls, texts, and other notifications over the network 260. For example, other applications 216 may be email or instant messaging applications that receive predictive result messages from the server 230. Other applications 216 may include device interfaces and other display modules that can receive input and / or output information. For example, other applications 216 may include processor-executable software programs for asset management, including a graphical user interface (GUI) configured to provide the user 240 with an interface for viewing code program output.

[0028] The user device 210 may further include a database 218 stored in the temporary and / or non-temporary memory of the user device 210, which stores various applications and data and can be used while various modules of the user device 210 are running. The database 218 may store a user profile of the user 240, predictions previously viewed or saved by the user 240, historical data received from the server 230, and / or similar. In some embodiments, the database 218 may be local to the user device 210. However, in other embodiments, the database 218 may be external to the user device 210 and accessible by the user device 210, such as a cloud storage system and / or database accessible via the network 260.

[0029] The user device 210 includes at least one network interface component 217 adapted to communicate with the data vendor server 245 and / or server 230. In various embodiments, the network interface component 217 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet® device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency, infrared, Bluetooth®, and short-range wireless communication devices.

[0030] The data vendor server 245 may correspond to a server that hosts one or more of the databases 203a-n (or applications 216 collectively called 203) to provide the server 230 with training datasets containing input-output (e.g., description-code) examples. The databases 203 may be implemented by one or more relational databases, distributed databases, cloud databases, and / or similar.

[0031] The data vendor server 245 includes at least one network interface component 226 adapted to communicate with user devices 210 and / or server 230. In various embodiments, the network interface component 226 may include various other types of wired and / or wireless network communication devices, including DSL (e.g., Digital Subscriber Line) modems, PSTN (Public Switched Telephone Network) modems, Ethernet® devices, broadband devices, satellite devices, and / or microwave, radio frequency, infrared, Bluetooth®, and short-range wireless communication devices. For example, in one implementation, the data vendor server 245 may transmit asset information from database 203 to server 230 via the network interface 226.

[0032] Server 230 may house the program synthesis module 130 and its submodules as described in Figure 1. In some implementations, the program synthesis module 130 may receive data from the database 219 of the data vendor server 245 via the network 260 and generate a synthesized code program. The generated code program may also be sent to a user device 210 via the network 260 for review by the user 240.

[0033] For example, exemplary datasets obtained from data vendor servers 245, 270, and 280 may include THEPILE, BIGQUERY, and BIGPYTHON. The natural language dataset THEPILE is an 825.18 GiB English text corpus collected for language modeling, as described in Gao et al., The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. This dataset is built from 22 diverse, high-quality subsets, one of which is programming language data collected from GitHub repositories with over 100 Starts, which constitute 7.6% of the dataset. Since the majority of THEPILE is English text, the resulting model is called the Natural Language CODEGEN Model (CODEGEN-NL).

[0034] As another example, the multilingual dataset BIGQUERY is a subset of the BigQuery dataset published by Google, consisting of code in multiple programming languages ​​(under open-source licenses). For multilingual training, the following six programming languages ​​are selected: C, C++, Go, Java®, JavaScript®, and Python. Therefore, models trained on BIGQUERY are called multilingual CODEGEN models (CODEGEN-MULTI).

[0035] As another example, the monolingual dataset BIG PYTHON contains a large amount of data for the programming language Python. In October 2021, it was compiled from publicly available, non-personalized Python code licensed permissively from GitHub. As a result, models trained on BIG PYTHON are called monolingual CODEGEN models (CODEGEN-MONO).

[0036] Therefore, the program synthesis module 130 can train a family of CODEGEN models in various sizes of Transformer models with 350M, 2.7B, 6.1B, and 16.1B parameters. The first three configurations allow for direct comparison with open-source large-scale language models GPT®-NEO (350M, 2.7B) and GPT-J120 (6B) trained on text corpora.

[0037] The database 232 may be stored in the temporary and / or non-temporary memory of the server 230. In one implementation, the database 232 may store data obtained from the data vendor server 245. In one implementation, the database 232 may store parameters of the program synthesis model 130. In one implementation, the database 232 may store previously generated code programs, benchmarks, and / or similar.

[0038] In some embodiments, the database 232 may be local to the server 230. However, in other embodiments, the database 232 may be external to the server 230, such as a cloud storage system and / or database accessible via the network 260, and accessible by the server 230.

[0039] Server 230 includes at least one network interface component 233 adapted to communicate with user devices 210 and / or data vendor servers 245, 270, or 280 via network 260. In various embodiments, the network interface component 233 may include various other types of wired and / or wireless network communication devices, including DSL (e.g., Digital Subscriber Line) modems, PSTN (Public Switched Telephone Network) modems, Ethernet® devices, broadband devices, satellite devices, and / or microwave, radio frequency (RF), and infrared (IR) communication devices.

[0040] Network 260 can be implemented as a single network or a combination of multiple networks. For example, in various embodiments, network 260 may include the Internet or one or more intranets, a terrestrial communication line network, a wireless network, and / or other suitable types of networks. Thus, network 260 can correspond to small-scale communication networks, such as private or local area networks, or large-scale networks, such as wide area networks or the Internet, accessible by various components of system 200.

[0041] Figure 3 is an exemplary logic flow diagram illustrating a method for code program synthesis for a target problem based on the framework shown in Figures 1 and 2, according to several embodiments described herein. One or more of the processes of Method 300 may be implemented in the form of executable code stored on a non-temporary tangible machine-readable medium, which, when executed at least partially by one or more processors (e.g., a TPU with high-speed toroidal mesh interconnects), can cause one or more processors to execute one or more of the processes. In some embodiments, Method 300 corresponds to the operation of a program synthesis module 130 (e.g., Figures 1 and 2) that performs program synthesis for a target problem.

[0042] In step 301, a natural language specification of the task problem may be received, for example, via a communication interface (e.g., 115 in Figure 1). For example, as shown in Figure 5, the task problem specification may be "extracting the user name of an email address".

[0043] In one implementation, the specification can be broken down into multiple prompts (see, for example, p1-p5 in Figure 5). In one implementation, the specification is paired with the expected results of the task problem, for example, sample code subprograms s1-s5 in Figure 5. The generated results are compared with the expected results to evaluate the accuracy of one or more language models.

[0044] In step 302, during the first generation time step, a first prompt may be obtained from multiple prompts (see, for example, p1 "import re and define a regular expression that matches an email address" in Figure 5).

[0045] In step 304, in response to a first prompt in a first generation time step, one or more language models may generate a first subprogram (see, for example, s1 in Figure 5). For example, one or more language models include one or more autoregressive Transformer models trained for the purpose of next token prediction language modeling.

[0046] In some implementations, the program synthesis model may generate one or more questions in natural language to further clarify the user intent. This may occur prior to the first generation timestamp or at the first or any subsequent generation timestamp. The user may provide a response in natural language to further clarify the user intent in program generation. The user's response may be incorporated into the next generation timestep so that the next subprogram may be generated conditionally on the user's response.

[0047] In step 306, during the second generation time step, a second prompt from multiple prompts (see, for example, p2 in Figure 5) may be obtained.

[0048] In step 308, one or more language models generate a second subprogram (see, for example, s2 in Figure 5) in a second generation time step in response to a second prompt and based on the first subprogram and the first prompt. In some implementations, if a user response for clarification is received, the second subprogram may be generated as an additional condition of that user response.

[0049] In step 310, the subprograms generated from a series of generation time steps corresponding to multiple prompts (e.g., s1 to s5 in Figure 5) can be concatenated into an output code program.

[0050] In step 312, the output code program can be executed to generate the result of the task problem (see, for example, 530 in Figure 5).

[0051] Figure 4 is an exemplary logic flow diagram illustrating a method for training the program synthesis module 130 shown in Figures 1 and 2 in parallel on multiple processors, according to some embodiments described herein. One or more of the processes of Method 400 may be implemented in the form of executable code stored on a non-temporary tangible machine-readable medium, which, when executed at least partially by one or more processors (e.g., a TPU with a high-speed toroidal mesh interconnect), can cause one or more processors to execute one or more of the processes. In some embodiments, Method 400 corresponds to the operation and training of the program synthesis module 130 (e.g., Figures 1 and 2) that performs program synthesis for a target problem. For example, as described with respect to Figures 1 and 2, the program synthesis module 130 may comprise one or more language models that are trained on one or more hardware processors (e.g., processor 101 in Figure 1) by performing the same computation in parallel on different input data on different hardware processors.

[0052] For example, TPU hardware with high-speed toroidal mesh interconnects can enable efficient parallel processing. To efficiently utilize the hardware, model training is implemented in JAX. The pjit() operator is used for parallel evaluation in JAX. This operator enables a paradigm called SPMD (single-program, multiple-data) code, which refers to a parallel processing technique in which the same computation is executed in parallel for different input data on different devices. Specifically, pjit() is an API exposed to the JAX XLA SPMD partitioner that allows a given function to be evaluated in parallel with equivalent semantics across the logical mesh of computation.

[0053] In step 402, a training data set including pairs of text descriptions and corresponding code program snippets can be received.

[0054] In step 404, a batch of training data can be partitioned into multiple subsets of the training data.

[0055] In step 406, multiple subsets of the training data can be distributed to multiple virtual machines of one or more hardware processors, respectively. For example, the library JAXFORMER can recruit a designated coordinator node to orchestrate a cluster of TPU-VMs using a custom TCP / IP protocol. For data parallel processing, the coordinator partitions the batch and partitions that partition to individual TPU-VMs.

[0056] In one implementation, for model parallel processing, two schemes for sharding model parameters are supported: (1) Intra-TPU-VM where parameters are sharded across MXU cores within a physical TPU-v4 board and replicated across the board, (2) Inter-TPU-VM where parameters are sharded across TPU-v4 boards and activations are replicated. Both the intra-TPU-VM sharding scheme and the inter-TPU-VM sharding scheme are implemented based on a specific pjit() logical mesh specification (r, p, c), where r is the number of parameter replications, p is the number of parameter partitions, c is the number of logical cores per board across n c number of TPU boards having n b number of logical cores per board, and d × p = n b and r × p × c = n b × n c is made to be so.

[0057] In step 408, the program synthesis module 130 may determine whether the total size of one or more language models is greater than a predefined threshold. For example, the intra-TPU-VM scheme is employed for models with a size of 6B parameters or less, where the total size of the model and optimizer parameters fits into the combined HBM memory of a single TPU-v4 board.

[0058] In response to determining in step 410 that the total size of one or more language models is less than or equal to a predefined threshold, method 400 may proceed to step 412, where the total amount of model parameters is replicated across several TPU boards. For example, a TPU-v4-512 slice with nb=64 and nc=4 is configured as (r,p,c)=(64,1,4). That is, the parameters are replicated across 64 boards with p=1 total inter-board partition and c=4 in-board parallel processing across logic chips. In this configuration, the mean gradient is accumulated across boards via with_sharding_constraint(), effectively emulating the behavior of the xmap() operator.

[0059] In response to determining in step 410 that the total size of one or more language models is greater than a predefined threshold, method 400 may proceed to step 414, where modules share the total amount of model parameters across several TPU boards. For example, the inter-TPU-VM scheme is employed for models exceeding 6B parameters in size, where the model and optimizer parameters need to be sharded across TPU-v4 boards. For example, a TPU-v4-512 slice with nb=64 and nc=4 is configured as (r,p,c)=(1,64,4). For larger slices such as a TPU-v4-1024 with nb=128, redundancy can be introduced into the parameter sharding, for example, as (r,p,c)=(2,64,4). In this configuration, activations are replicated across boards via with_sharding_constraint(). Furthermore, (r,p,c) enables backward compatibility for logical hardware layout migration from TPU-v3 with c=8 to TPU-v4 with c=4 by adjusting p without requiring resharding.

[0060] Method 400 then proceeds to step 416, in which the module continues to train the language model in parallel on different hardware processors. For example, the Adam optimizer (described in Kingma et al., Adam: A method for stochastic optimization, in proceedings of International Conference on Learning Representation (Poster), 2015) may be used for training with parameters (β1,β2,∈)=(0.9,0.999,1e-08) and global gradient norm clipping of 1.0. The learning rate function over time may follow GPT-3® (Brown et al., Language models are few-shot learners, Advances in neural information processing systems, 33:1877-1901, 2020), which includes a warm-up step and cosine annealing.

[0061] Figure 5 is a simplified block diagram showing an example of program synthesis via multi-turn conversation according to embodiments described herein. Illustrative Example 500 demonstrates a multi-turn programming benchmark that performs the task of extracting usernames from email addresses. For example, a natural language description of the task can be broken down into a series of prompts p1-p5 that are executed in multiple turns, as shown by the designed discourse 501.

[0062] In generation 502, each turn consists of a prompt and a response generated by one or more language models. Prompts p1-p5 can include templates that accept various test cases (i.e., {input} for different task descriptions). For example, five test cases may be provided for each problem, each test case including (a) an input 505 that replaces {input} in the prompt before the prompt is supplied to the model, and (b) an expected output 510 that the output of the generated program is compared to in order to evaluate its functional correctness. In the example shown 500, input 505 is a string containing abc.xyz@example.com, which replaces {input} in p2, and the expected output is abc xyz. The model is conditional on the concatenation of interleaved past prompts and generated responses.

[0063] In generation 502, in each term, the language model generates subprograms s1 to s5 based on prompts p1 to p5 and previously generated subprograms. For example, in turn 1, subprogram s1 is generated using prompt p1 as input. In turn 2, subprogram s2 is generated using the concatenated inputs p1, s1, and p2. The process may continue until all five prompts p1 to p5 have been processed. The resulting subprograms s1 to s5 from each turn are then concatenated to form the final output code program 515.

[0064] In evaluation 503, the output code program 515 is executed to generate the actual output result 520, which is then compared with the expected output 510.

[0065] During training, the input training data may include a problem description and the expected output 510 for that problem. Therefore, the training objective (e.g., cross-entropy) can be calculated by comparing the actual output 520 generated by the language model with the expected output 510. The language model can then be updated using the training objective. Data experiment example

[0066] In one embodiment, functional accuracy (as described in Hendrycks et al., Measuring coding challenge competence with APPS, in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021) is used to evaluate the model for program synthesis. Evaluation is performed using single-turn and multi-turn benchmarks. Both benchmarks are handwritten to avoid data leakage from the training data. The model is evaluated using both the single-turn evaluation benchmark and the proposed multi-turn benchmark.

[0067] HumanEval is a benchmark for measuring a model's program synthesis ability in Python® (MIT license). It includes 164 handwritten programming problems. Each problem provides a prompt with a description of the function to be generated, the function signature, and an example of a test case in assertion form. The model must complete the function given by the prompt so that it can pass all the provided test cases. Since the user intent is specified in a single prompt and provided to the model only once, evaluations on HumanEval are considered single-turn evaluations, distinguishing them from multi-turn evaluations where the user intent is broken down into multiple steps. We sample 158 nucleus at top-p (p=0.95) according to the sampling strategy described by Holtzman et al., The curious case of neural text degeneration, in the minutes of the 2020 International Conference on Learning Representation.

[0068] Model 130 is compared to the Codex model described in Chen et al., Evaluating large language models trained on code, arXiv preprint arXiv:2107.03374, 2021, which demonstrates state-of-the-art performance in HumanEval. Furthermore, program-synthesized model 130 is compared to the open-source large language models GPT NEO (Black et al., GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021) and GPT-J (Wang et al., GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, May 2021). These models were trained on THEPILE (described in Gao et al., The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020) and are therefore similar to the CODEGEN-NL models of this application in terms of training data and model size. All models were evaluated at temperatures t∈{0.2,0.6,0.8}, and pass@k was calculated for each model at k∈{1,10,100}. For direct comparison with the results of Chen et al., Evaluating large language models trained on code, arXiv preprint arXiv:2107.03374, 2021, the temperature that yields the best-performing pass@k is selected for each k. The results for the program-synthesized model 130 and baseline are summarized in Figure 6.

[0069] As shown in Figure 6, the CODEGEN-NL models (350M, 2.7B, 6.1B) perform better than or at least as well as the respective GPT-NEO and GPT-J models. Further training CODEGEN-NL with multilingual programming language data (BIGQUERY) yields CODEGEN-MULTI. The multilingual CODEGEN model performs significantly better than the models trained on THEPILE (GPT-NEO, GPT-J, CODEGEN-NL). CODEGEN MULTI is then fine-tuned on a Python-only dataset (BIGPYTHON) to obtain CODEGEN-MONO. Program synthesis capabilities are significantly improved. Therefore, Python program synthesis capabilities increase as the amount of Python training data increases. For almost all models, as expected, increasing the model size improves overall performance. The Python-monolingual CODEGEN model performs comparably to or better than the current state-of-the-art model, Codex. CODEGEN-MONO 2.7B performs worse than CODEX 2.5B when k=100, but performs better when k∈{1,10}. Although only half the size of CODEX 12B, CODEGEN-MONO 6.1B shows a pass@k score close to that of the highest-performing Codex, CODEX 12B. The present application's CODEGEN-MONO 16.1B performs even better.

[0070] The success of a program synthesis system largely depends on how well it understands user intent. If the system is based on a language model, the perplexity of the problem prompts provides a proxy for the system to understand the user intent specification. A low perplexity of an intent specification under a given model indicates that this intent specification is compatible with the knowledge learned by the model from the training data. Data experiments are then conducted to investigate whether better prompt understanding leads to more functionally accurate programs, using lower prompt perplexities as proxies. All problems are partitioned into pass and non-pass problems. A pass problem is one in which at least one of 200 samples passes all test cases, while a non-pass problem is one in which none of the 200 samples pass all test cases. Based on samples from the CODEGEN-MONO model, the average perplexity of the problem prompts for pass problems and the average perplexity of the problem prompts for non-pass problems are calculated. The results are shown in Figure 7. The prompts for pass problems have lower perplexity than the prompts for non-pass problems. This finding means that when user intent specifications are better understood by the model, program synthesis is more likely to succeed. In fact, some training data includes interleaved sequences of natural language comments and programs, where the comments describe the functionality of the subsequent programs. Therefore, it is hypothesized that user intent specifications similar to such patterns will be better understood by the model, and thus lead to better program synthesis. Thus, user intents can be specified in multiple turns so that the model can focus on partial problems at a time, which makes it easier for the model to understand the user intents.

[0071] In some embodiments, the process of synthesizing a program is a conversational program synthesis paradigm in which the process is cast as a multi-turn conversation between the user and the system, as illustrated in Figure 5. To validate such a paradigm, the Multi-Turn Programming Benchmark (MTPB) is developed. The MTPB consists of 115 problems written by experts. The description for solving each problem is broken down into multiple steps, each step including a description in natural language (prompts). To solve the problem, the model must synthesize a functionally correct subprogram, (1) following the description in the current step, and (2) taking into account the description and synthesized subprograms in the previous step (e.g., correct back-referencing of functions and / or variables defined in the previous step).

[0072] In one embodiment, the MTPB can be constructed by queuing a set of 115 problems requiring a diverse range of programming knowledge, including problems involving mathematics, array operations, string manipulation, algorithms, data science, and other knowledge. For each problem, there is a multi-turn prompt that serves as step-by-step instructions to the model, ensuring that (1) it forms at least three turns and (2) problem solving does not belong to a single turn. Five test cases are included for each problem to evaluate the synthetic program with functional accuracy. When creating the test cases, non-trivial test cases are included that pass only if the correct solution is given. Unlike HumanEval, where the model is expected to complete a partially defined function, the MTPB problems only provide prompts, so the model must generate the solution from the outset. Free-form generation may allow for more potential solutions, but the lack of an entry point providing test case inputs makes it difficult to test the generated code on a variety of test cases.221 To overcome this challenge, test case inputs are embedded within the prompts. Specifically, prompts are written as Python-formatted string8, and input values ​​are assigned to variable names when a particular test case is applied to the problem. For example, combining the prompt "Define a string named 's' with the value {var}." with the test case input var = 'Hello' will format it as "Define a string named 's' with the value 'Hello'."

[0073] In one embodiment, for execution, a history of prompt-and-generated completion pairs is concatenated into a self-contained program (see the example shown in Figure 5). The program is then executed in an isolated Python environment according to the single-turn HumanEval benchmark. However, since the HumanEval problem is constructed so that known function signatures are completed, the calls to the generated code under a set of functional unit tests are trivial. In the case of multi-turn problems, there is no guarantee that such entry points (or return values) will be generated. To avoid the problem of missing return signatures (or return values), the final prompt of a multi-turn problem in the MTPB is always specified to print out the resulting state to the terminal. The benchmark execution environment then overloads the Python print(args) function and stores args on the stack. If the sampled code for the final prompt of the problem does not contain a print() statement, which is a valid convention for printing to the terminal in Python or specifically in a Jupyter notebook, the AST of the generated code is modified to insert a call to print(). Finally, type-relaxed equivalence checks (e.g., implicit conversions between lists and tuples) are performed on the args against the predefined gold output of the problem to determine whether the test fails or succeeds.

[0074] This analysis involves further data experiments to investigate how model size and data size affect program synthesis capabilities in a multi-turn conversational paradigm. Language models are trained on the following datasets in four sizes: 350M, 2.7B, 6.1B, and 16.1B: THEPILE, BIGQUERY, and BIGPYTHON, which increasingly include Python data (see Section 2.1 for further details). GPT-NEO, GPT-J, and CODEGEN-NL models are trained on THEPILE. CODEGEN-MULTI models are initialized with CODEGEN-NL models and then trained on BIGQUERY. CODEGEN-MONO models are initialized with CODEGEN-MULTI models and then trained on BIGPYTHON. In MTPB, each problem has five test cases, and 40 samples are sampled for each test case for each model, and the pass rate for each problem is calculated based on these samples. The MTPB evaluation results (average pass rate) for the CODEGEN models and baseline are shown in Figure 8. Clearly, performance on MTPB has been demonstrated as a function of model size and data size. This suggests that the ability to synthesize conversational programs scales as a function of model size and data size. The model is simply trained for autoregressive language modeling purposes. As the model and data scale up, conversational ability, i.e., the ability to synthesize programs in a multi-turn manner, emerges.

[0075] Multi-turn decomposition improves the model's understanding of user intent specifications, which leads to improved program synthesis capabilities. To test this hypothesis, we concatenated each specification into a single-turn equivalent of the multi-turn specification. We compare the perplexity of multi-turn prompts with the perplexity of the concatenated single-turn prompts under four CODEGEN-MONO models. The average perplexity across all problems in the MTPB is shown in the left panel of Figure 9. For all models, single-turn specifications have a higher average perplexity than multi-turn specifications. This means that multi-turn user specifications can be better understood by the model. The average perplexity for both multi-turn and single-turn intent specifications in larger models is slightly lower than in smaller models, indicating that larger models understand user intent better than smaller ones.

[0076] Next, program synthesis is compared between the pass rate with multi-turn prompts and the pass rate with concatenated single-turn prompts. The results are shown in the right panel of Figure 9. Multi-turn specifications are nearly 10 percentage points or more higher than single-turn specifications for all model sizes. Combined with the perplexity analysis described above, it is thought that by breaking down user specifications into multiple steps and leveraging the emerging conversational capabilities of large language models, user specifications can be digested more easily and programs can be synthesized more effectively.

[0077] Furthermore, the problems were classified by difficulty based on their average pass rates (less than 30% for "difficult," and more than 70% for "easy"), allowing us to examine the interaction effect between difficulty and model size on improvements through multi-turn decomposition. The results are shown in Figure 10. Across almost all model sizes and difficulties, multi-turn prompts yielded significant improvements over single-turn prompts, with most improvements being approximately 10 percent points or more. Interestingly, larger models (6.1B and 16.1B) were invariant to multi-turn decomposition for easy problems (see the two short bars 0.19% and -0.25% in Figure 10). This means that specification decomposition is unnecessary or pointless when the problem can be easily understood by the model (due to the combined effect of the ease of the problem and the higher capability of the larger model). This is in fact consistent with the motivational assumption of the present application that decomposing complex specifications makes the problem easier to understand and improves program synthesis.

[0078] This description and accompanying drawings illustrating aspects, embodiments, implementations, or applications of the present invention should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational modifications can be made without departing from the spirit and scope of this specification and the claims. In some cases, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure the embodiments of this disclosure. Similar numbers in two or more figures represent the same or similar elements.

[0079] This description provides specific details illustrating several embodiments consistent with the present disclosure. Numerous specific details are provided to provide a complete understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be implemented without some or all of these specific details. The specific embodiments disclosed herein are illustrative but not limiting. Those skilled in the art may recognize other elements not specifically described herein but within the scope and spirit of the disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in relation to one embodiment may be incorporated into other embodiments unless otherwise specifically described or unless the one or more features would render the embodiment non-functional.

[0080] While illustrative embodiments have been shown and described, the foregoing disclosures are subject to extensive modifications, alterations, and substitutions, and in some cases, certain features of the embodiments may be adopted without corresponding use of other features. Those skilled in the art will recognize numerous variations, substitutions, and modifications. Therefore, the scope of the present invention should be limited only by the following claims, which should be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A system for synthesizing code programs for task problems, A user interface configured to receive a natural language description of the task problem from the user, Memory for storing one or more language models and multiple processor-executable instructions, One or more hardware processors and The system includes, and the one or more hardware processors read the plurality of processor-executable instructions, Decomposing the aforementioned natural language description into a first prompt that describes at least a first subprogram and a second prompt that describes a second subprogram, The first subprogram is generated in response to the first prompt using one or more language models, The first subprogram is generated in response to the second prompt, using one or more language models as conditions, The first subprogram and the second subprogram are combined into an output code program, The output code program is executed in the code environment, thereby producing the result of the task problem. A system that performs actions including those mentioned above.

2. The aforementioned operation is, The one or more language models described above generate questions that require clarification of the user intent for the task problem, The user interface is used to receive user responses in response to the question. The system according to claim 1, further comprising:

3. The system according to claim 2, wherein one or more of the first subprogram and the second subprogram are generated by the one or more language models, further based on the user response.

4. The system according to claim 2, wherein the natural language description is a training sample, and the one or more language models are trained on the basis of a training objective of comparing the result of the task problem with an expected result paired with the training sample.

5. The system according to claim 1, wherein the one or more hardware processors include tensor processing units (TPUs) having high-speed toroidal mesh interconnects.

6. The system according to claim 5, wherein the one or more language models are trained on the one or more hardware processors, and the training includes performing the same computation in parallel on different hardware processors for different input data.

7. The aforementioned operation is, Partitioning a batch of training data into multiple subsets of training data, Distributing multiple subsets of the training data to multiple virtual machines of one or more hardware processors, respectively. The system according to claim 6, further comprising:

8. The aforementioned operation is, In response to determining that the total size of one or more language models is less than or equal to a predefined threshold, the total amount of model parameters is replicated across several TPU boards. In response to the determination that the total size of one or more language models is greater than the predefined threshold, the total amount of the model parameters is shared across several TPU boards. The system according to claim 7, further comprising:

9. A method for synthesizing code programs for task problems, The steps include receiving a natural language description of the task problem from the user via a user interface, The steps include: using one or more hardware processors to decompose the natural language description into a first prompt for describing at least a first subprogram and a second prompt for describing a second subprogram; The steps include generating the first subprogram in response to the first prompt using one or more language models implemented on the one or more hardware processors, The steps include generating a second subprogram in response to a second prompt, with the first subprogram as a condition, using one or more language models implemented on one or more hardware processors, The steps include combining the first subprogram and the second subprogram into an output code program, A step of executing the output code program in the code environment, thereby producing the result of the task problem. A method that includes this.

10. A non-temporary processor-readable storage medium for storing multiple processor-executable instructions for code program synthesis for a task problem, wherein the instructions are: The user interface receives a natural language description of the task problem from the user, Decomposing the aforementioned natural language description into a first prompt that describes at least a first subprogram and a second prompt that describes a second subprogram, To generate the first subprogram in response to the first prompt using one or more language models, The first subprogram is generated in response to the second prompt, using one or more language models as conditions, The first subprogram and the second subprogram are combined into an output code program, The output code program is executed in the code environment, thereby producing the result of the task problem. A non-temporary processor-readable storage medium that is executed by one or more hardware processors to perform operations including the following.

Images