System and method for an interactive framework for program synthesis

Abstract

The embodiments described herein provide a program synthesis framework that generates code programs through a multi-turn conversation between a user and a system. Specifically, a description to solve a target problem is decomposed into multiple steps, each step including a description in natural language (prompt) that is input to a generative model as a user utterance. The model then synthesizes a functionally correct sub-program according to the current user utterance and taking into account the description and the synthesized sub-programs in the previous steps. The sub-programs generated in the multiple steps are then combined to form the output of a program that responds to the target problem.

Description

[Technical Field]

[0001] This disclosure is a non-provisional application of and claims priority to U.S. Provisional Application No. 63 / 317,244, filed March 7, 2022, which claims priority to U.S. Patent Application No. 17 / 889,998, filed August 17, 2022, which is expressly incorporated herein by reference in its entirety.

[0002] [Technical field] The present disclosure relates generally to machine learning models and automatic code generation, and more particularly to pre-training machine learning models for conversational program synthesis. [Background technology]

[0003] Program synthesis is the automatic generation of a program in a programming language (e.g., Python, C++, Pearl), often via machine learning models, that satisfies a user intent expressed in some form of specification (e.g., in natural language). Successful program synthesis increases the productivity of experienced programmers and makes programming accessible to a much broader range of users. However, long-term challenges in program synthesis remain unmet. For example, maintaining an expressive search space often requires a large search space of programming code. Thus, efficiently exploring a large number of candidates in a huge search space remains difficult.

[0004] As another example, providing an adequate specification of user intent for code generation remains difficult. User intent for program synthesis can be a logical specification specifying logical relationships between program inputs and outputs, input-output examples, or natural language. Since compiling the specification alone can sometimes require an amount of effort comparable to that of program generation itself, providing a complete formal or informal specification requires significant effort from the user and defeats the purpose of program synthesis. However, simpler specifications, such as those using mere input-output examples or abstract descriptions in natural language, can underspecify the problem and lead to undesirable solutions.

[0005] Therefore, there is a need for a program synthesis technique that addresses these issues. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a simplified diagram of a computing device for implementing an interactive program synthesis system according to some embodiments. [Figure 2] FIG. 2 is a simplified block diagram of a networked system suitable for implementing the program synthesis framework depicted in FIG. 1 and other embodiments described herein. [Figure 3] FIG. 3 is an exemplary logic flow diagram illustrating a method for code program synthesis for a target problem based on the framework shown in FIGS. 1-2, according to some embodiments described herein. [Figure 4] FIG. 3 is an exemplary logic flow diagram illustrating a method for training the program synthesis module shown in FIGS. 1-2 in parallel on multiple processors, according to some embodiments described herein. [Figure 5] FIG. 2 is a simplified block diagram illustrating an example of program composition via multi-turn conversation, according to embodiments described herein. [Figure 6]6 is an exemplary data performance chart illustrating exemplary performance of the conversational program synthesis module illustrated in FIGS. 1-5, according to embodiments described herein. [Figure 7] 6 is an exemplary data performance chart illustrating exemplary performance of the conversational program synthesis module illustrated in FIGS. 1-5, according to embodiments described herein. [Figure 8] 6 is an exemplary data performance chart illustrating exemplary performance of the conversational program synthesis module illustrated in FIGS. 1-5, according to embodiments described herein. [Figure 9] 6 is an exemplary data performance chart illustrating exemplary performance of the conversational program synthesis module illustrated in FIGS. 1-5, according to embodiments described herein. [Figure 10] 6 is an exemplary data performance chart illustrating exemplary performance of the conversational program synthesis module illustrated in FIGS. 1-5, according to embodiments described herein.

[0007] In the figures, elements with the same name have the same or similar function. DETAILED DESCRIPTION OF THE INVENTION

[0008] As used herein, the term "network" may include any hardware or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or therewith.

[0009] As used herein, the term "module" may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0010] In light of the challenges of efficient search and user intent specification in program synthesis, the embodiments described herein provide a program synthesis framework that generates code programs through a multi-turn conversation between a user and a system. Specifically, a description for solving a target problem is decomposed into multiple steps, each of which includes a natural language (prompt) description that is input to a generative model as a user utterance. The model then synthesizes a functionally correct subprogram according to the current user utterance and taking into account the description and synthesized subprograms from previous steps. The subprograms generated in the multiple steps are then combined to form a program output that responds to the target problem.

[0011] In another embodiment, the generated program may also be inaccurate when the code generation model has high uncertainty. The program synthesis framework may generate questions to clarify the user intent. In this way, the user's answers to the system's questions may be used as the user intent.

[0012] In one embodiment, user intent for program synthesis is used as a specification specifying logical relationships between program inputs and outputs, input-output examples, or natural language. Because most programmers do not develop a comprehensive vision of their intent until well into the programming process, an interactive program synthesis approach may be employed. In this approach, the user interacts with the synthesis system by incrementally providing specifications in natural language while receiving responses from the system in the form of synthesized subprograms, allowing the user and system to interactively complete the program in a multi-turn conversation.

[0013] In one embodiment, the program synthesis framework can be built on a Transformer model that is used to capture dependencies between sequence elements through an attention mechanism. For example, a large-scale autoregressive transformer is trained on a large Python dataset powered by cloud tensor processing units (TPUs).

[0014] In one embodiment, due to the large size of the Transformer model for program synthesis, a parallel processing technique is employed to improve training efficiency, allowing the same calculation to be performed in parallel on different devices with different input data. Specifically, for data parallelism, batches of training data are partitioned and distributed to individual TPU virtual machines (VMs). For model parallelism, two schemes for sharding model parameters are supported: (1) intra-TPU-VM, where parameters are sharded across matrix unit (MXU) cores within a physical TPU board and replicated across the board; and (2) inter-TPU-VM, where parameters are sharded across TPU-v4 boards and activations are replicated.

[0015] 1 is a simplified diagram of a computing device for implementing an interactive program synthesis system, according to some embodiments. As shown in FIG. 1, computing device 100 includes a processor 110 coupled to memory 120. The operation of computing device 100 is controlled by processor 110. While computing device 100 is shown with one processor 110, it is understood that processor 110 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), and / or the like within computing device 100. Computing device 100 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0016] Memory 120 may be used to store software executed by computing device 100 and / or one or more data structures used during operation of computing device 100. Memory 120 may include one or more types of machine-readable media. Some common forms of machine-readable media may include a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read by a processor or computer.

[0017] Processor 110 and / or memory 120 may be arranged in any suitable physical arrangement. In some embodiments, processor 110 and / or memory 120 may be implemented on the same substrate, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), and / or the like. In some embodiments, processor 110 and / or memory 120 may comprise distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, processor 110 and / or memory 120 may be located in one or more data centers and / or cloud computing facilities.

[0018] In some examples, memory 120 may include a non-transitory, tangible, machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 110), may cause the one or more processors to perform methods described in further detail herein. For example, as shown, memory 120 includes instructions for program synthesis module 130, which may be used to implement and / or emulate systems and models and / or to implement any of the methods described further herein. In some examples, online adaptation module 130 may receive input 140, such as natural language input specifying user intent, for example, via data interface 115. Program synthesis module 130 may generate program snippet output 150 in response to input 140.

[0019] For example, the program synthesis module 130 may include a bidirectional t-transformer encoder 131 and an autoregressive transformer decoder 132 for learning conventional next-token prediction language modeling. Therefore, the architecture of the program synthesis module 130 may follow a Transformer decoder with left-to-right causal masking. The encoder 131 may employ a rotational position embedding for position encoding (for further details on position encoding, see Su et al., Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021). For the forward pass, the encoder 131 and decoder 132 may include a self-attention circuit and a feedforward circuit in parallel to improve communication overhead. Specifically, the module output may be generated according to the following: x t+1 =x t +attn(ln(x t ))+mlp(ln(x t )) This allows us to simultaneously implement the computation of self-attention attn( ) and feedforward mlp( ) using the layer norm ln( ). During implementation, the architecture and hyperparameter choices were optimized specifically for the hardware layout of the TPU-v4.

[0020] In some examples, program synthesis module 130 may be implemented using hardware, software, and / or a combination of hardware and software.

[0021] Some examples of computing devices, such as computing device 100, may include non-transitory tangible machine-readable media containing executable code that, when executed by one or more processors (e.g., processor 110), may cause the one or more processors to perform the processes of a method. Some common forms of machine-readable media that may contain the processes of a method are, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, a punch card, paper tape, any other physical medium with a pattern of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read by a processor or a computer.

[0022] FIG. 2 is a simplified block diagram of a networked system suitable for implementing the program synthesis framework described in FIG. 1 and other embodiments described herein. In one embodiment, block diagram 200 illustrates a system including a user device 210 that may be operated by a user 240, data vendor servers 245, 270, and 280, a server 230, and other forms of devices, servers, and / or software components that operate to perform various methods according to the described embodiments. Exemplary devices and servers may include devices that may be similar to computing device 100 described in FIG. 1, standalone, and enterprise-class servers that run an OS such as the MICROSOFT® OS, UNIX® OS, LINUX® OS, or other suitable device- and / or server-based OS. It can be understood that the devices and / or servers illustrated in FIG. 2 may be deployed in other manners, and that the operations performed and / or services provided by such devices and / or servers may be combined or separated for a given embodiment, or may be performed by a greater or fewer number of devices and / or servers. One or more of the devices and / or servers may be operated and / or maintained by the same or different entities.

[0023] User device 210, data vendor servers 245, 270, and 280, and server 230 may communicate with each other via network 260. User device 210 may be utilized by user 240 (e.g., driver, system administrator, etc.) to access various features available on user device 210, which may include processes and / or applications associated with server 230 to receive output data anomaly reports.

[0024] User device 210, data vendor server 245, and server 230 may each include one or more processors, memory, and other suitable components for executing instructions, such as program code and / or data stored on one or more computer-readable media, to implement the various applications, data, and steps described herein. For example, such instructions may be stored on one or more computer-readable media, such as memory or data storage devices, internal and / or external to the various components of system 200 and / or accessible via network 260.

[0025] User device 210 may be implemented as a communications device that may utilize appropriate hardware and software configured for wired and / or wireless communications with data vendor server 245 and / or server 230. For example, in one embodiment, user device 210 may be implemented as an autonomous vehicle, a personal computer (PC), a smartphone, a laptop / tablet computer, a wristwatch with appropriate computing hardware resources, glasses with appropriate computing hardware (e.g., GOOGLE GLASS®), other types of wearable computing devices, embedded communications devices, and / or an APPLE® or IPAD®, or other type of computing device capable of transmitting and / or receiving data. Although only one communications device is shown, multiple communications devices may function similarly.

[0026] 2 includes a user interface (UI) application 212 and / or other applications 216, which may correspond to executable processes, procedures, and / or applications with associated hardware. For example, the user device 210 may receive a message from the server 230 indicating a synthesized code program and display the message via the UI application 212. In other embodiments, the user device 210 may include additional or different modules with dedicated hardware and / or software, as needed.

[0027] In various embodiments, user device 210 includes other applications that may be desired in particular embodiments to provide functionality to user device 210. For example, other applications 216 may include security applications for implementing client-side security features, programmatic client applications for interfacing with appropriate application programming interfaces (APIs) over network 260, or other types of applications. Other applications 216 may also include communication applications, such as email, texting, voice, social networking, and IM applications, that allow a user to send and receive emails, calls, texts, and other notifications over network 260. For example, other applications 216 may be email or instant messaging applications that receive predicted outcome messages from server 230. Other applications 216 may include device interfaces and other display modules that may receive input and / or output information. For example, other applications 216 may include a software program for asset management, executable by a processor, that includes a graphical user interface (GUI) configured to provide user 240 with an interface for viewing code program output.

[0028] The user device 210 may further include a database 218 stored in temporary and / or non-transitory memory of the user device 210 that stores various applications and data and may be utilized during the execution of various modules of the user device 210. The database 218 may store a user profile for the user 240, predictions previously viewed or saved by the user 240, historical data received from the server 230, and / or the like. In some embodiments, the database 218 may be local to the user device 210. However, in other embodiments, the database 218 may be external to and accessible by the user device 210, such as a cloud storage system and / or database accessible via the network 260.

[0029] User device 210 includes at least one network interface component 217 adapted to communicate with data vendor server 245 and / or server 230. In various embodiments, network interface component 217 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency, infrared, Bluetooth, and near field communication devices.

[0030] Data vendor server 245 may correspond to a server hosting one or more of databases 203a-n (or applications 216, collectively referred to as 203) to provide training data sets including example input-output (e.g., description-code, etc.) examples to server 230. Database 203 may be implemented by one or more relational databases, distributed databases, cloud databases, and / or the like.

[0031] Data vendor server 245 includes at least one network interface component 226 adapted to communicate with user device 210 and / or server 230. In various embodiments, network interface component 226 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency, infrared, Bluetooth, and near field communication devices. For example, in one implementation, data vendor server 245 may transmit asset information from database 203 to server 230 via network interface 226.

[0032] 1 and its sub-modules. In some implementations, the program synthesis module 130 may receive data from the database 219 of the data vendor server 245 via the network 260 to generate a synthesized code program. The generated code program may also be transmitted to the user device 210 via the network 260 for review by the user 240.

[0033] For example, exemplary datasets retrieved from data vendor servers 245, 270, and 280 may include THEPILE, BIGQUERY, and BIGPYTHON. The natural language dataset THEPILE is an 825.18 GiB English text corpus collected for language modeling as described in Gao et al., *The pile: An 800gb dataset of diverse text for language modeling*. arXiv preprint arXiv:2101.00027, 2020. This dataset is constructed from 22 diverse, high-quality subsets, one of which is programming language data collected from a GitHub repository with over 100 Starts, which constitutes 7.6% of the dataset. Because the majority of THEPILE is English text, the resulting model is called the Natural Language CODEGEN model (CODEGEN-NL).

[0034] As another example, the multilingual dataset BIGQUERY is a subset of Google's publicly available BigQuery dataset, which consists of code in multiple programming languages ​​(under open-source licenses). For multilingual training, the following six programming languages ​​are selected: C, C++, Go, Java, JavaScript, and Python. Therefore, a model trained on BIGQUERY is called a multilingual CODEGEN model (CODEGEN-MULTI).

[0035] As another example, the monolingual dataset BIGPYTHON contains a large amount of data for the programming language Python. It is compiled from public, non-personal data from GitHub, consisting of permissively licensed Python code, in October 2021. The resulting model trained on BIGPYTHON is called the monolingual CODEGEN model (CODEGEN-MONO).

[0036] Thus, program synthesis module 130 may train a family of CODEGEN models with various sizes of Transformer models with 350M, 2.7B, 6.1B, and 16.1B parameters. The first three configurations allow direct comparison with the open-source large-scale language models GPT-NEO (350M, 2.7B) and GPT-J120 (6B), both of which are trained on text corpora.

[0037] The database 232 may be stored in temporary and / or non-temporary memory of the server 230. In one implementation, the database 232 may store data obtained from the data vendor server 245. In one implementation, the database 232 may store parameters of the program synthesis model 130. In one implementation, the database 232 may store previously generated code programs, benchmarks, and / or the like.

[0038] In some embodiments, database 232 may be local to server 230. However, in other embodiments, database 232 may be external to and accessible by server 230, such as a cloud storage system and / or database accessible via network 260.

[0039] Server 230 includes at least one network interface component 233 adapted to communicate with user device 210 and / or data vendor server 245, 270, or 280 over network 260. In various embodiments, network interface component 233 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency (RF), infrared (IR) communication devices.

[0040] Network 260 may be implemented as a single network or a combination of multiple networks. For example, in various embodiments, network 260 may include the Internet or one or more intranets, landline networks, wireless networks, and / or other suitable types of networks. Thus, network 260 may correspond to a small communications network, such as a private or local area network, or a larger network, such as a wide area network or the Internet, accessible by various components of system 200.

[0041] 3 is an exemplary logic flow diagram illustrating a method for code program synthesis for a target problem based on the framework shown in FIGS. 1-2, according to some embodiments described herein. One or more of the processes of method 300 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors (e.g., TPUs with high-speed toroidal mesh interconnects), may cause the one or more processors to perform one or more of the processes. In some embodiments, method 300 corresponds to the operation of program synthesis module 130 (e.g., FIGS. 1-2) that performs program synthesis for a target problem.

[0042] In step 301, a natural language specification of a task problem may be received, for example, via a communication interface (e.g., 115 in FIG. 1). For example, as shown in FIG. 5, the task problem specification may be "extracting the user name of an email address."

[0043] In one implementation, the specification may be broken down into multiple prompts (e.g., see p1-p5 in FIG. 5). In one implementation, the specification is paired with expected results of the task problem, e.g., sample code subprograms s1-s5 in FIG. 5. The generated results are compared to the expected results to assess the accuracy of one or more language models.

[0044] In step 302, in a first generation time step, a first prompt from a plurality of prompts (see, for example, p1 "import re and define a regular expression that matches an email address" in FIG. 5) may be obtained.

[0045] In step 304, a first subprogram (e.g., see s1 in FIG. 5 ) may be generated by one or more language models in response to a first prompt in a first generation time step. For example, the one or more language models may include one or more autoregressive Transformer models trained with a next-token prediction language modeling objective.

[0046] In some implementations, the program composition model may generate one or more questions in natural language to further clarify the user intent. This may occur prior to the first generation timestamp or at the first or any subsequent generation timestamp. The user may provide a natural language response to further clarify the user intent in the program generation. The user-provided response may be incorporated into the next generation timestep such that the next subprogram may be generated conditional on the user-provided response.

[0047] In step 306, in a second generation time step, a second prompt from the plurality of prompts (see, for example, p2 in FIG. 5) may be obtained.

[0048] In step 308, a second sub-program (e.g., see s2 in FIG. 5 ) is generated in a second generation time step by one or more language models in response to the second prompt and based on the first sub-program and the first prompt. In some implementations, if a clarifying user response is received, the generation of the second sub-program may be further conditional on the user response.

[0049] In step 310, the generated sub-programs from a series of generation time steps corresponding to multiple prompts (eg, s1-s5 in FIG. 5) may be concatenated into an output code program.

[0050] In step 312, the output code program may be executed to generate a task problem result (see, for example, 530 in FIG. 5).

[0051] FIG. 4 is an exemplary logic flow diagram illustrating a method for training program synthesis module 130 shown in FIGS. 1-2 in parallel on multiple processors, according to some embodiments described herein. One or more of the processes of method 400 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors (e.g., a TPU with a high-speed toroidal mesh interconnect), may cause the one or more processors to perform one or more of the processes. In some embodiments, method 400 corresponds to the operation and training of program synthesis module 130 (e.g., FIGS. 1-2) that performs program synthesis for a target problem. For example, as described with respect to FIGS. 1-2, program synthesis module 130 may comprise one or more language models that are trained on one or more hardware processors (e.g., processor 101 of FIG. 1) by performing the same calculations on different input data in parallel on different hardware processors.

[0052] For example, TPU hardware with high-speed toroidal mesh interconnects can enable efficient parallel processing. To efficiently utilize the hardware, model training is implemented in JAX. Parallel evaluation in JAX employs the pjit() operator. This operator enables a paradigm called SPMD (single-program, multiple-data) code, which refers to a parallel processing technique in which the same computation is executed in parallel on different devices with different input data. Specifically, pjit() is an API exposed to JAX's XLA SPMD partitioner, which allows a given function to be evaluated in parallel with equivalent semantics across a logical mesh of computations.

[0053] In step 402, a training data set may be received that includes pairs of text descriptions and corresponding code program snippets.

[0054] In step 404, the batch of training data may be partitioned into multiple subsets of training data.

[0055] In step 406, multiple subsets of the training data may be distributed across multiple virtual machines on one or more hardware processors. For example, the library JAXFORMER may recruit a designated coordinator node to orchestrate a cluster of TPU-VMs using a custom TCP / IP protocol. For data parallelism, the coordinator partitions the batches and partitions the partitions to individual TPU-VMs.

[0056] In one implementation, two schemes for sharding model parameters are supported for model parallelism: (1) Intra-TPU-VM, where parameters are sharded across MXU cores within a physical TPU-v4 board and replicated across the board; and (2) Inter-TPU-VM, where parameters are sharded across TPU-v4 boards and activations are replicated. Both the intra-TPU-VM and inter-TPU-VM sharding schemes are implemented based on a specific pjit() logical mesh specification (r, p, c), where r is the number of parameter replicas, p is the number of parameter partitions, and c is the number of n partitions, respectively. c n logic cores b is the number of logical cores per board across TPU boards, where d × p = n b and r×p×c=n b ×n c It is designed to be like this.

[0057] In step 408, the program synthesis module 130 may determine whether the total size of one or more language models is greater than a predefined threshold. For example, the intra-TPU-VM scheme may be employed for models whose size is 6B parameters or less, and the total amount of model and optimizer parameters fits into the combined HBM memory of a single TPU-v4 board.

[0058] In response to determining at step 410 that the total size of one or more language models is less than or equal to a predefined threshold, method 400 may proceed to step 412, where the total amount of model parameters is replicated across several TPU boards. For example, a TPU-v4-512 slice with nb=64 and nc=4 is configured as (r,p,c)=(64,1,4). That is, parameters are replicated across r=64 boards, with p=1 total inter-board partitions and intra-board parallelism across c=4 logic chips. In this configuration, average gradients are accumulated across boards via with_sharding_constraint(), effectively emulating the behavior of the xmap() operator.

[0059] In response to determining in step 410 that the total size of one or more language models is greater than a predefined threshold, method 400 may proceed to step 414, where the module shares the total amount of model parameters across several TPU boards. For example, an inter-TPU-VM scheme is adopted for models greater than 6B parameters in size, where model and optimizer parameters need to be sharded across TPU-v4 boards. For example, a TPU-v4-512 slice with nb = 64 and nc = 4 is configured as (r, p, c) = (1, 64, 4). For larger slices, such as a TPU-v4-1024 with nb = 128, redundancy may be introduced in the parameter sharding, e.g., (r, p, c) = (2, 64, 4). In this configuration, activations are replicated across boards via with_sharding_constraint(). Furthermore, (r, p, c) enables backward compatibility for logical hardware layout migration from TPU-v3 with c=8 to TPU-v4 with c=4 by adjusting p without requiring resharding.

[0060] Method 400 then proceeds to step 416, where the modules continue training the language models in parallel on different hardware processors. For example, an Adam optimizer (described in Kingma et al., “Adam: A method for stochastic optimization,” in proceedings of International Conference on Learning Representation (Poster), 2015) may be employed for training with parameters (β1, β2, ∈) = (0.9, 0.999, 1e-08) and global gradient norm clipping of 1.0. The learning rate function over time may follow GPT-3 (described in Brown et al., “Language models are few-shot learners,” Advances in neural information processing systems, 33:1877-1901, 2020) with a warm-up step and cosine annealing.

[0061] 5 is a simplified block diagram illustrating an example of program composition through multi-turn conversation, according to an embodiment described herein. Illustrative example 500 shows a multi-turn programming benchmark that performs the task of extracting usernames from email addresses. For example, a natural language description of the task can be decomposed into a series of prompts p1-p5 that are executed in multiple turns, as shown by designed discourse 501.

[0062] In generation 502, each turn consists of a prompt and a generated response by one or more language models. Prompts p1-p5 can include templates that accept various test cases (i.e., {input} for different task descriptions). For example, for each problem, five test cases can be provided, each including (a) an input 505 that replaces the {input} in the prompt before the prompt is fed to the model, and (b) an expected output 510 against which the generated program output is compared to assess its functional correctness. In the example shown 500, input 505 is a string containing abc.xyz@example.com, which replaces {input} in p2, and the expected output is abc xyz. The model is conditioned on the concatenation of interleaved past prompts and generated responses.

[0063] In generation 502, at each term, the language model generates subprograms s1-s5 based on prompts p1-p5 and previously generated subprograms. For example, in turn 1, subprogram s1 is generated using prompt p1 as input. In turn 2, subprogram s2 is generated using the concatenated input of p1, s1, and p2. The process can continue until all five prompts p1-p5 have been processed. The resulting subprograms s1-s5 in each turn are then concatenated to form the final output code program 515.

[0064] In evaluation 503 , an output code program 515 is executed to generate actual output results 520 that can be compared to the expected output 510 .

[0065] During training, the input training data may include a problem description and the expected output 510 for the problem. Thus, a training objective (e.g., cross-entropy) may be calculated by comparing the actual output 520 generated from the language model with the expected output 510. The training objective may then be used to update the language model. Data experiment example

[0066] In one embodiment, functional accuracy (as described in Hendrycks et al., Measuring coding challenge competence with APPS, in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021) is used to evaluate models for program synthesis. Evaluation is performed on single-turn and multi-turn benchmarks. Both benchmarks are handwritten to avoid data leakage from the training data. We evaluate the models on both the single-turn evaluation benchmark and the proposed multi-turn benchmark.

[0067] HumanEval is a benchmark for measuring a model's program synthesis capabilities in Python (MIT license). It contains 164 handwritten programming problems. Each problem provides a prompt with a description of the function to be generated, the function signature, and an example test case in the form of an assertion. The model must complete the function given the prompt so that it can pass all provided test cases. Because the user intent is specified in a single prompt and provided to the model once, evaluations on HumanEval are considered single-turn evaluations, which are distinct from multi-turn evaluations in which the user intent is decomposed into multiple steps. We sampled 158 nuclei with top-p (p=0.95) following the sampling strategy described in Holtzman et al., "The curious case of neural text degeneration," in Proceedings of the 2020 International Conference on Learning Representation.

[0068] Model 130 is compared to the Codex model described in Chen et al., Evaluating large language models trained on code, arXiv preprint arXiv:2107.03374, 2021, which shows state-of-the-art performance in HumanEval. Additionally, program synthesis model 130 is compared to the open-source large-scale language models GPT NEO (Black et al., GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021) and GPT-J (Wang et al., GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, May 2021). These are trained on THEPILE (described in Gao et al., The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020), and are therefore similar to our CODEGEN-NL model in terms of training data and model size. All models are evaluated at temperatures t ∈ {0.2, 0.6, 0.8}, and pass@k is calculated for each model at k ∈ {1, 10, 100}. For direct comparison with the results of Chen et al., Evaluating large language models trained on code, arXiv preprint arXiv:2107.03374, 2021, the temperature that yields the best pass@k performance for each k is selected. The results of the program synthesis model 130 and the baseline are summarized in Figure 6.

[0069] As shown in Figure 6, the CODEGEN-NL model (350M, 2.7B, 6.1B) outperforms or is comparable to the respective GPT-NEO and GPT-J models. Further training of CODEGEN-NL on multilingual programming language data (BIGQUERY) yields CODEGEN-MULTI. The multilingual CODEGEN model significantly outperforms models trained on TEPILE (GPT-NEO, GPT-J, CODEGEN-NL). CODEGEN MULTI is then fine-tuned on a Python-only dataset (BIGPYTHON) to yield CODEGEN-MONO. Program synthesis capability is significantly improved. Thus, Python program synthesis capability increases with increasing amounts of Python training data. For nearly all models, as expected, increasing model size improves overall performance. The Python-monolingual CODEGEN model performs competitively or even better than the current state-of-the-art model, Codex. CODEGEN-MONO 2.7B performs worse than CODEX 2.5B when k = 100, but performs better when k ∈ {1, 10}. Although only half the size of CODEX 12B, CODEGEN-MONO 6.1B achieves a pass@k score close to that of CODEX 12B, the best-performing Codex. Our CODEGEN-MONO 16.1B outperforms it.

[0070] The success of a program synthesis system depends heavily on how well it understands user intent. When a system is based on a language model, the perplexity of problem prompts provides a proxy for the system's understanding of user intent specifications. A low perplexity of an intent specification under a given model indicates that the intent specification is compatible with the knowledge learned by the model from training data. Data experiments are then conducted to examine whether better prompt understanding, using lower prompt perplexity as a proxy, leads to more functionally accurate programs. All problems are partitioned into pass and non-pass. A pass problem is one in which at least one sample out of 200 passes all test cases, while a non-pass problem is one in which none of the 200 samples passes all test cases. Based on samples from the CODEGEN-MONO model, the average perplexity of the pass problem prompts is calculated, and the average perplexity of the non-pass problem prompts is calculated. The results are shown in Figure 7. Pass problem prompts have lower perplexity than non-pass problem prompts. This finding implies that program synthesis is more likely to succeed if the user intent specification is better understood by the model. Indeed, some training data contains interleaved sequences of natural language comments and programs, where the comments describe the functionality of the subsequent program. Therefore, it is speculated that user intent specifications similar to such patterns will be better understood by the model and thus lead to better program synthesis. Therefore, user intent may be specified in multiple turns so that the model can focus on partial problems at a time, making it easier for the model to understand the user intent.

[0071] In some embodiments, a conversational program synthesis paradigm is used in which the process of synthesizing a program is cast as a multi-turn conversation between a user and a system, similar to the one illustrated in FIG. 5. To validate such a paradigm, a multi-turn programming benchmark (MTPB) is developed. The MTPB consists of 115 problems written by experts. The description for solving each problem is decomposed into multiple steps, and each step includes a description in natural language (a prompt). To solve a problem, the model must (1) follow the description in the current step and (2) synthesize a functionally correct subprogram given the description and synthesized subprograms in previous steps (e.g., correct backreferences to functions and / or variables defined in previous steps).

[0072] In one embodiment, the MTPB can be constructed by curating a set of 115 problems requiring a diverse range of programming knowledge, including problems requiring mathematical problems, array operations, string manipulation, algorithms, data science, and other knowledge. For each problem, multi-turn prompts are included to serve as step-by-step instructions for the model, ensuring that (1) there are at least three turns and (2) the problem solution does not reside in a single turn. Five test cases are included for each problem to evaluate the synthesized program for functional correctness. When creating test cases, non-trivial test cases are included that pass only if the correct solution is given. Unlike HumanEval, where the model is expected to complete a partially defined function, MTPB problems only provide prompts, so the model must generate a solution from scratch. While free-form generation may allow for a larger number of potential solutions, the lack of an entry point to provide test case inputs makes it difficult to test the generated code on a diverse set of test cases. To overcome this challenge, test case inputs are embedded within the prompts. Specifically, prompts are written in Python formatted string8, and the input value is assigned to a variable name when the particular test case is applied to the problem. For example, the prompt "Define a string named 's' with the value {var}." combined with the test case input var = 'Hello' will be formatted as "Define a string named 's' with the value 'Hello'."

[0073] In one embodiment, for execution, the history of pairs of prompts and generated completions is concatenated into a self-contained program (see the example shown in Figure 5). The program is then executed in an isolated Python environment according to the single-turn HumanEval benchmark. However, since HumanEval problems are constructed to complete known function signatures, invoking the generated code under a set of functional unit tests is trivial. In the multi-turn case, such an entry point (or return value) is not guaranteed to be generated. To avoid the problem of missing return signatures (or return values), the final prompt of a multi-turn problem in MTPB is always specified to print out the resulting state to the terminal. The benchmark execution environment then overloads the Python print(args) function and stores args on the stack. If the code sampled for the problem's final prompt does not contain a print() statement, which is a valid convention for printing on the terminal in Python, or specifically in Jupyter notebooks, the AST of the generated code is modified to insert a print() call. Finally, a type-relaxed equivalence check (e.g., implicit conversions between lists and tuples) of the args is performed against the predefined gold output of the problem to determine the failure or success of the test.

[0074] This analysis further conducts data experiments to examine how model size and data size affect program synthesis capabilities in a multi-turn conversational paradigm. The language models are trained on the following datasets with four sizes: 350M, 2.7B, 6.1B, and 16.1B: THEPILE, BIGQUERY, and BIGPYTHON, which contain increasingly large amounts of Python data (see Section 2.1 for further details). The GPT-NEO, GPT-J, and CODEGEN-NL models are trained on THEPILE. The CODEGEN-MULTI model is initialized with the CODEGEN-NL model and then trained on BIGQUERY. The CODEGEN-MONO model is initialized with the CODEGEN-MULTI model and then trained on BIGPYTHON. In MTPB, each problem has five test cases, and each model samples 40 samples for each test case, based on which the pass rate for each problem is calculated. The MTPB evaluation results (average pass rate) for the CODEGEN model and the baseline are shown in Figure 8. Clearly, performance on MTPB is demonstrated as a function of model size and data size. This suggests that the ability of conversational program synthesis scales as a function of model size and data size. The model is simply trained for autoregressive language modeling purposes. As the model and data scale up, conversational capabilities, i.e., the ability to synthesize programs in a multi-turn fashion, emerge.

[0075] Multi-turn decomposition improves the model's understanding of user intent specifications, leading to improved program synthesis capabilities. To test this hypothesis, we concatenated each specification into a single turn to create a single-turn counterpart for the multi-turn specifications. We compare the perplexity of multi-turn prompts with the perplexity of concatenated single-turn prompts under the four CODEGEN-MONO models. The average perplexity across all problems in the MTPB is shown in the left panel of Figure 9. For all models, single-turn specifications have a higher average perplexity than multi-turn specifications. This means that multi-turn user specifications can be better understood by the model. The average perplexity for both multi-turn and single-turn intent specifications in larger models is slightly lower than that in smaller models, indicating that larger models understand user intent better than smaller models.

[0076] Program synthesis is then compared for pass rates with multi-turn prompts versus concatenated single-turn prompts. The results are shown in the right panel of Figure 9. Multi-turn specifications perform nearly or more than 10 percentage points better than single-turn specifications for all model sizes. Combined with the perplexity analysis above, we believe that decomposing user specifications into multiple steps and leveraging the emergent conversational capabilities of large language models will make user specifications more easily digestible and programs more successfully synthesized.

[0077] Furthermore, problems were classified by difficulty level based on their average pass rate (less than 30% is "hard" and over 70% is "easy") to examine the interaction effect between difficulty and model size on the improvement due to multi-turn decomposition. The results are shown in Figure 10. Across almost all model sizes and difficulties, multi-turn prompts yielded significant improvements over single-turn prompts, with most improvements being near or above 10 percentage points. Interestingly, larger models (6.1B and 16.1B) were invariant to multi-turn decomposition for easy problems (see the two short bars in Figure 10, 0.19% and -0.25%). This implies that if a problem can be easily understood by a model (due to the combined effect of problem ease and the higher power of larger models), decomposing the specification is unnecessary or unhelpful. This is indeed consistent with our motivating assumption that decomposing complex specifications facilitates problem understanding and improves program synthesis.

[0078] This description and the accompanying drawings illustrating aspects, embodiments, implementations, or applications of the present invention should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of the specification and claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the embodiments of the present disclosure. Like numbers in two or more figures represent the same or similar elements.

[0079] In this description, specific details are set forth to describe some embodiments consistent with the present disclosure. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative, not limiting. Those skilled in the art may recognize other elements not specifically described herein that are within the scope and spirit of the present disclosure. Additionally, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments, unless specifically described otherwise or unless one or more features render the embodiment non-functional.

[0080] While illustrative embodiments have been shown and described, the foregoing disclosure contemplates a wide range of modifications, changes, and substitutions, and in some instances, some features of the embodiments may be employed without the corresponding use of other features. Those skilled in the art will recognize many variations, substitutions, and modifications. Accordingly, the scope of the present invention is to be limited only by the claims that follow, and it is appropriate that such claims be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A system for code program synthesis for a task problem, comprising: a communications interface configured to receive a natural language specification of the task problem; wherein the specification is broken down into a plurality of prompts; a memory storing one or more language models and a plurality of processor-executable instructions; one or more hardware processors; wherein the one or more hardware processors read the plurality of processor-executable instructions to: obtaining a first prompt from the plurality of prompts at a first generation time step; generating a first subprogram in response to the first prompt at the first generation time step with the one or more language models; obtaining a second prompt from the plurality of prompts at a second generation time step; generating, with the one or more language models, a second sub-program in response to the second prompt and based on the first sub-program and the first prompt at the second generation time step; concatenating generated subprograms from a series of generated time steps corresponding to the plurality of prompts into an output code program; generating a result of the task problem by executing the output code program; and A system that performs operations including:

2. 10. The system of claim 1, wherein the specification is paired with an expected outcome of the task problem, and the generated results are compared to the expected results to assess accuracy of the one or more language models.

3. 10. The system of claim 1, wherein the one or more language models comprise one or more autoregressive Transformer models trained with a next-token prediction language modeling objective.

4. 10. The system of claim 1, wherein the one or more hardware processors comprise tensor processing units (TPUs) with high-speed toroidal mesh interconnects.

5. The system of claim 4 , wherein the one or more language models are trained on the one or more hardware processors by performing the same calculation on different input data in parallel on different hardware processors.

6. The operation is Partitioning the batch of training data into multiple subsets of training data; distributing a plurality of subsets of the training data to a plurality of virtual machines of the one or more hardware processors, respectively; The system of claim 5 further comprising:

7. The operation is replicating a total amount of model parameters across several TPU boards in response to determining that a total size of the one or more language models is less than or equal to a predefined threshold; in response to determining that a total size of the one or more language models is greater than the predefined threshold, sharing the total amount of model parameters across the number of TPU boards; The system of claim 6 further comprising:

8. A method for code program synthesis for a task problem, comprising: receiving, via a communications interface, a natural language specification of the task problem, the specification being broken down into a plurality of prompts; obtaining a first prompt from the plurality of prompts at a first generation time step; generating a first subprogram in response to the first prompt at the first generation time step using one or more language models implemented on one or more hardware processors; obtaining a second prompt from the plurality of prompts at a second generation time step; generating, with the one or more language models, a second sub-program in response to the second prompt and based on the first sub-program and the first prompt during the second generation time step; concatenating generated subprograms from a series of generated time steps corresponding to the plurality of prompts into an output code program; generating a result of the task problem by executing the output code program; A method comprising:

9. 9. The method of claim 8, wherein the one or more language models are trained on the one or more hardware processors by performing the same calculations on different input data in parallel on different hardware processors.

10. Partitioning the batch of training data into a plurality of subsets of training data; distributing a plurality of subsets of the training data to a plurality of virtual machines of the one or more hardware processors, respectively; replicating a total amount of model parameters across several TPU boards in response to determining that a total size of the one or more language models is less than or equal to a predefined threshold; in response to determining that a total size of the one or more language models is greater than the predefined threshold, sharing the total amount of model parameters across the number of TPU boards; 10. The method of claim 9, further comprising:

Images