System and method for an interactive framework for program synthesis

Abstract

The embodiments described herein provide a program synthesis framework that generates code programs through a multi-turn conversation between a user and a system. Specifically, a description to solve a target problem is decomposed into multiple steps, each step including a description in natural language (prompt) that is input to a generative model as a user utterance. The model then synthesizes a functionally correct sub-program according to the current user utterance and taking into account the description and the synthesized sub-programs in the previous steps. The sub-programs generated in the multiple steps are then combined to form the output of a program that responds to the target problem.

Description

[Technical field]

[0001] This disclosure is a non-provisional application of, and claims priority to, U.S. Provisional Application No. 63 / 317,244, filed March 7, 2022, which claims priority to U.S. Patent Application No. 17 / 889,998, filed August 17, 2022, the entire contents of which are expressly incorporated herein by reference.

[0002] [Technical field] The present disclosure relates generally to machine learning models and automatic code generation, and more specifically to pre-training machine learning models for conversational program synthesis. [Background technology]

[0003] Program synthesis is the automatic generation, often via machine learning models, of a program in a programming language (e.g., Python, C++, Pearl) that satisfies a user intent expressed in some form of specification (e.g., in a natural language). Successful program synthesis increases the productivity of experienced programmers and makes programming accessible to a much broader set of users. However, long-term challenges in program synthesis remain unmet. For example, a large search space of programming code is often required to maintain an expressive search space. Thus, efficiently exploring a large number of candidates in a large search space remains difficult.

[0004] As another example, it remains difficult to provide an adequate specification of user intent for code generation. User intent for program synthesis can be a logical specification specifying logical relations between the inputs and outputs of the program, input-output examples, or natural language. Since compiling the specification alone may even require an amount of effort comparable to that of the program generation itself, providing a complete formal or informal specification requires a great deal of effort from the user, which defeats the purpose of program synthesis. However, simpler specifications, such as those using just input-output examples or abstract descriptions in natural language, may underspecify the problem and lead to undesirable solutions.

[0005] Therefore, there is a need for a program synthesis technique that addresses these issues. [Brief description of the drawings]

[0006] [Figure 1] 1 is a simplified diagram of a computing device for implementing an interactive program synthesis system according to some embodiments. [Diagram 2] FIG. 2 is a simplified block diagram of a networked system suitable for implementing the program synthesis framework depicted in FIG. 1 and other embodiments described herein. [Diagram 3] FIG. 3 is an exemplary logic flow diagram illustrating a method for code program synthesis for a target problem based on the framework illustrated in FIGS. 1-2, according to certain embodiments described herein. [Figure 4] FIG. 3 is an exemplary logic flow diagram illustrating a method for training the program synthesis module shown in FIGS. 1-2 in parallel on multiple processors according to some embodiments described herein. [Diagram 5] FIG. 2 is a simplified block diagram illustrating an example of program composition via multi-turn conversation, according to embodiments described herein. [Figure 6]6 is an exemplary data performance chart illustrating an exemplary performance of the conversational program synthesis module described in FIGS. 1-5, according to embodiments described herein. [Figure 7] 6 is an exemplary data performance chart illustrating an exemplary performance of the conversational program synthesis module described in FIGS. 1-5, according to embodiments described herein. [Figure 8] 6 is an exemplary data performance chart illustrating an exemplary performance of the conversational program synthesis module described in FIGS. 1-5, according to embodiments described herein. [Figure 9] 6 is an exemplary data performance chart illustrating an exemplary performance of the conversational program synthesis module described in FIGS. 1-5, according to embodiments described herein. [Figure 10] 6 is an exemplary data performance chart illustrating an exemplary performance of the conversational program synthesis module described in FIGS. 1-5, according to embodiments described herein.

[0007] In the figures, elements with the same name have the same or similar functions. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] As used herein, the term "network" may include any hardware or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or in conjunction therewith.

[0009] As used herein, the term "module" may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0010] In view of the challenges in efficient search and user intent specification in program synthesis, the embodiments described herein provide a program synthesis framework that generates code programs through a multi-turn conversation between a user and a system. Specifically, a description to solve a target problem is decomposed into multiple steps, each step including a description in natural language (prompt) that is input to a generative model as a user utterance. The model then synthesizes a functionally correct sub-program according to the current user utterance and taking into account the description and the synthesized sub-programs in the previous steps. The sub-programs generated in the multiple steps are then combined to form the output of a program that responds to the target problem.

[0011] In another embodiment, the generated program may also be inaccurate when the code generation model has high uncertainty. The program synthesis framework may generate questions to clarify the user intent. In this way, the user's answers to the system's questions may be used as the user intent.

[0012] In one embodiment, user intent for program synthesis is used as a specification that specifies logical relationships between the inputs and outputs of a program, input-output examples, or natural language. Because most programmers do not build a comprehensive vision of their intent until well into the programming process, an interactive program synthesis approach may be employed. In this approach, the user interacts with the synthesis system by incrementally providing specifications in natural language while receiving responses from the system in the form of synthesized subprograms, allowing the user and system to interactively complete the program in a multi-turn conversation.

[0013] In one embodiment, the program synthesis framework can be built on a Transformer model that is used to capture dependencies between sequence elements through an attention mechanism. For example, a large-scale autoregressive transformer is trained on a large Python dataset powered by cloud tensor processing units (TPUs).

[0014] In one embodiment, due to the large size of the Transformer model for program synthesis, a parallel processing technique is adopted to improve training efficiency, which allows the same calculation to be performed in parallel on different input data on different devices. Specifically, for data parallel processing, batches of training data are partitioned and distributed to individual TPU virtual machines (VMs). For model parallel processing, two schemes for sharding of model parameters are supported: (1) intra-TPU-VM, where parameters are sharded across matrix unit (MXU) cores in a physical TPU board and replicated across boards; (2) inter-TPU-VM, where parameters are sharded across TPU-v4 boards and activations are replicated.

[0015] 1 is a simplified diagram of a computing device for implementing an interactive program synthesis system, according to some embodiments. As shown in FIG. 1, computing device 100 includes a processor 110 coupled to a memory 120. The operation of computing device 100 is controlled by processor 110. Although computing device 100 is shown with one processor 110, it is understood that processor 110 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs) and / or the like within computing device 100. Computing device 100 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0016] The memory 120 may be used to store software executed by the computing device 100 and / or one or more data structures used during operation of the computing device 100. The memory 120 may include one or more types of machine-readable media. Some common forms of machine-readable media may include a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read by a processor or a computer.

[0017] The processor 110 and / or memory 120 may be arranged in any suitable physical arrangement. In some embodiments, the processor 110 and / or memory 120 may be implemented on the same substrate, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), and / or the like. In some embodiments, the processor 110 and / or memory 120 may comprise distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 110 and / or memory 120 may be located in one or more data centers and / or cloud computing facilities.

[0018] In some examples, memory 120 may include a non-transitory tangible machine-readable medium including executable code that, when executed by one or more processors (e.g., processor 110), may cause the one or more processors to perform methods described in further detail herein. For example, as shown, memory 120 includes instructions for program synthesis module 130, which may be used to implement and / or emulate systems and models and / or to implement any of the methods described further herein. In some examples, online adaptation module 130 may receive input 140, such as, for example, a natural language input specifying a user intent, via data interface 115. Program synthesis module 130 may generate output 150 of a program snippet in response to input 140.

[0019] For example, the program synthesis module 130 may include a bidirectional t-transformer encoder 131 and an autoregressive transformer decoder 132 for learning regular next token prediction language modeling. Thus, the architecture of the program synthesis module 130 may follow a Transformer decoder with left-to-right causal masking. The encoder 131 may employ a rotational position embedding for position encoding (further details of position encoding are described in Su et al., Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021). For the forward pass, the encoder 131 and the decoder 132 may include a self-attention circuit and a feedforward circuit in parallel to improve communication overhead. Specifically, the module output may be generated according to the following: x t+1 =x t +attn(ln(x t ))+mlp(ln(x t )) This allows us to simultaneously implement the computation of self-attention attn( ) and feedforward mlp( ) with layer norm ln( ). During the implementation, the choice of architecture and hyperparameters was optimized specifically for the hardware layout of the TPU-v4.

[0020] In some examples, program synthesis module 130 may be implemented using hardware, software, and / or a combination of hardware and software.

[0021] Some examples of computing devices, such as computing device 100, may include non-transitory tangible machine-readable media that includes executable code that, when executed by one or more processors (e.g., processor 110), may cause the one or more processors to perform the processes of a method. Some common forms of machine-readable media that may include the processes of a method are, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, a punch card, a paper tape, any other physical medium with a pattern of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read by a processor or a computer.

[0022] FIG. 2 is a simplified block diagram of a networked system suitable for implementing the program synthesis framework described in FIG. 1 and other embodiments described herein. In one embodiment, the block diagram 200 illustrates a system including a user device 210 that may be operated by a user 240, data vendor servers 245, 270, and 280, a server 230, and other forms of devices, servers, and / or software components that operate to perform various methods according to the described embodiments. Exemplary devices and servers may include devices that may be similar to the computing device 100 described in FIG. 1, standalone, and enterprise class servers that operate an OS such as the MICROSOFT® OS, UNIX® OS, LINUX® OS, or other suitable device and / or server-based OS. It can be understood that the devices and / or servers illustrated in FIG. 2 may be deployed in other manners, and that the operations performed and / or services provided by such devices and / or servers may be combined or separated for a given embodiment, and may be performed by a greater or lesser number of devices and / or servers. One or more of the devices and / or servers may be operated and / or maintained by the same or different entities.

[0023] The user device 210, data vendor servers 245, 270, and 280, and server 230 may communicate with each other via network 260. The user device 210 may be utilized by a user 240 (e.g., a driver, a system administrator, etc.) to access various features available on the user device 210, which may include processes and / or applications associated with the server 230 to receive output data anomaly reports.

[0024] User device 210, data vendor server 245, and server 230 may each include one or more processors, memory, and other suitable components for executing instructions, such as program code and / or data stored on one or more computer-readable media to implement the various applications, data, and steps described herein. For example, such instructions may be stored in one or more computer-readable media, such as memory or data storage devices, internal and / or external to the various components of system 200 and / or accessible via network 260.

[0025] User device 210 may be implemented as a communications device that may utilize appropriate hardware and software configured for wired and / or wireless communications with data vendor server 245 and / or server 230. For example, in one embodiment, user device 210 may be implemented as an autonomous vehicle, a personal computer (PC), a smartphone, a laptop / tablet computer, a watch with appropriate computing hardware resources, glasses with appropriate computing hardware (e.g., GOOGLE GLASS®), other types of wearable computing devices, embedded communications devices, and / or an APPLE® or IPAD®, or other types of computing devices capable of transmitting and / or receiving data. Although only one communications device is shown, multiple communications devices may function similarly.

[0026] 2 includes a user interface (UI) application 212 and / or other applications 216, which may correspond to executable processes, procedures, and / or applications having associated hardware. For example, the user device 210 may receive a message from the server 230 indicating a composed code program and display the message via the UI application 212. In other embodiments, the user device 210 may include additional or different modules having dedicated hardware and / or software, as appropriate.

[0027] In various embodiments, the user device 210 includes other applications that may be desired in a particular embodiment to provide functionality to the user device 210. For example, the other applications 216 may include security applications for implementing client-side security features, programmatic client applications for interfacing with appropriate application programming interfaces (APIs) over the network 260, or other types of applications. The other applications 216 may also include communication applications such as email, texting, voice, social networking, and IM applications that allow the user to send and receive emails, calls, texts, and other notifications over the network 260. For example, the other applications 216 may be email or instant messaging applications that receive predicted outcome messages from the server 230. The other applications 216 may include device interfaces and other display modules that may receive input and / or output information. For example, the other applications 216 may include software programs for asset management executable by the processor that include a graphical user interface (GUI) configured to provide the user 240 with an interface for viewing code program output.

[0028] The user device 210 may further include a database 218 stored in temporary and / or non-transient memory of the user device 210 that stores various applications and data and may be utilized during execution of various modules of the user device 210. The database 218 may store a user profile for the user 240, predictions previously viewed or saved by the user 240, historical data received from the server 230, and / or the like. In some embodiments, the database 218 may be local to the user device 210. However, in other embodiments, the database 218 may be external to and accessible by the user device 210, such as a cloud storage system and / or database accessible via the network 260.

[0029] The user device 210 includes at least one network interface component 217 adapted to communicate with the data vendor server 245 and / or the server 230. In various embodiments, the network interface component 217 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency, infrared, Bluetooth, and near field communication devices.

[0030] Data vendor server 245 may correspond to a server hosting one or more of databases 203a-n (or applications 216, collectively referred to as 203) to provide training data sets including input-output (e.g., description-code, etc.) examples to server 230. Database 203 may be implemented by one or more relational databases, distributed databases, cloud databases, and / or the like.

[0031] The data vendor server 245 includes at least one network interface component 226 adapted to communicate with the user device 210 and / or the server 230. In various embodiments, the network interface component 226 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency, infrared, Bluetooth, and near field communication devices. For example, in one implementation, the data vendor server 245 may transmit asset information from the database 203 to the server 230 via the network interface 226.

[0032] The server 230 may house the program synthesis module 130 described in Figure 1 along with its sub-modules. In some implementations, the program synthesis module 130 may receive data from the database 219 of the data vendor server 245 over the network 260 to generate a synthesized code program. The generated code program may also be transmitted over the network 260 to the user device 210 for review by the user 240.

[0033] For example, exemplary datasets obtained from data vendor servers 245, 270, 280 may include THEPILE, BIGQUERY, and BIGPYTHON. The natural language dataset THEPILE is an 825.18 GiB English text corpus collected as described in Gao et al., The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020 for language modeling. The dataset is constructed from 22 diverse and high-quality subsets, one of which is programming language data collected from a GitHub repository with over 100 Starts that constitutes 7.6% of the dataset. Because the majority of THEPILE is English text, the resulting model is called the natural language CODEGEN model (CODEGEN-NL).

[0034] As another example, the multilingual dataset BIGQUERY is a subset of the BigQuery dataset publicly available from Google and consists of code in multiple programming languages ​​(under open source licenses). For multilingual training, the following six programming languages ​​are selected: C, C++, Go, Java, JavaScript, and Python. Thus, the model trained on BIGQUERY is called the multilingual CODEGEN model (CODEGEN-MULTI).

[0035] As another example, the monolingual dataset BIGPYTHON contains a large amount of data in the programming language Python. It is compiled from public, non-personal data from GitHub consisting of permissively licensed Python code in October 2021. The resulting model trained on BIGPYTHON is called the monolingual CODEGEN model (CODEGEN-MONO).

[0036] Thus, the program synthesis module 130 may train a family of CODEGEN models in various sizes of Transformer models with 350M, 2.7B, 6.1B, and 16.1B parameters. The first three configurations allow direct comparison with the open source large-scale language models GPT-NEO (350M, 2.7B) and GPT-J120 (6B) trained on text corpora.

[0037] The database 232 may be stored in a temporary and / or non-temporary memory of the server 230. In one implementation, the database 232 may store data obtained from the data vendor server 245. In one implementation, the database 232 may store parameters of the program synthesis model 130. In one implementation, the database 232 may store previously generated code programs, benchmarks, and / or the like.

[0038] In some embodiments, database 232 may be local to server 230. However, in other embodiments, database 232 may be external to and accessible by server 230, such as a cloud storage system and / or database accessible via network 260.

[0039] Server 230 includes at least one network interface component 233 adapted to communicate with user device 210 and / or data vendor server 245, 270, or 280 over network 260. In various embodiments, network interface component 233 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency (RF), infrared (IR) communication devices.

[0040] Network 260 may be implemented as a single network or a combination of multiple networks. For example, in various embodiments, network 260 may include the Internet or one or more intranets, landline networks, wireless networks, and / or other suitable types of networks. Thus, network 260 may correspond to a small communications network, such as a private or local area network, or a larger network, such as a wide area network or the Internet, accessible by various components of system 200.

[0041] 3 is an exemplary logic flow diagram illustrating a method of code program synthesis for a target problem based on the framework illustrated in FIGS. 1-2, according to some embodiments described herein. One or more of the processes of method 300 may be implemented, at least in part, in the form of executable code stored on a non-transitory tangible machine-readable medium that, when executed by one or more processors (e.g., TPUs with high-speed toroidal mesh interconnects), may cause the one or more processors to execute one or more of the processes. In some embodiments, method 300 corresponds to the operation of program synthesis module 130 (e.g., FIGS. 1-2) performing program synthesis for a target problem.

[0042] In step 301, a natural language specification of a task problem may be received, for example, via a communication interface (e.g., 115 in FIG. 1). For example, as shown in FIG. 5, the task problem specification may be "extracting the user name of an email address."

[0043] In one implementation, the specification may be broken down into multiple prompts (e.g., see p1-p5 in FIG. 5). In one implementation, the specification is paired with expected outcomes of the task problem, e.g., sample code subprograms s1-s5 in FIG. 5. The generated results are compared to the expected results to assess the accuracy of one or more language models.

[0044] In step 302, in a first generation time step, a first prompt from a plurality of prompts (eg, see p1 “import re and define a regular expression that matches an email address” in FIG. 5) may be obtained.

[0045] In step 304, a first subprogram (e.g., see s1 in FIG. 5) may be generated by one or more language models in response to a first prompt in a first generation time step. For example, the one or more language models may include one or more autoregressive Transformer models trained with a next token prediction language modeling objective.

[0046] In some implementations, the program synthesis model may generate one or more questions in natural language to further clarify the user intent. This may occur prior to the first generation timestamp or at the first or any subsequent generation timestamp. The user may provide a response in natural language to further clarify the user intent in the program generation. The user-provided response may be incorporated into the next generation timestep such that the next subprogram may be generated conditional on the user-provided response.

[0047] In step 306, in a second generation time step, a second prompt from the plurality of prompts (eg, see p2 in FIG. 5) may be obtained.

[0048] In step 308, a second sub-program (e.g., see s2 in FIG. 5) is generated in a second generation time step by one or more language models in response to the second prompt and based on the first sub-program and the first prompt. In some implementations, the second sub-program may be generated further conditional on a clarification user response, if one is received.

[0049] In step 310, the generated sub-programs from a series of generation time steps corresponding to multiple prompts (eg, s1-s5 in FIG. 5) may be concatenated into an output code program.

[0050] In step 312, a task problem result (see, for example, 530 in FIG. 5) may be generated by executing the output code program.

[0051] FIG. 4 is an exemplary logic flow diagram illustrating a method for training the program synthesis module 130 shown in FIGS. 1-2 in parallel on multiple processors according to some embodiments described herein. One or more of the processes of the method 400 may be implemented, at least in part, in the form of executable code stored on a non-transitory tangible machine-readable medium that, when executed by one or more processors (e.g., a TPU with high-speed toroidal mesh interconnect), may cause the one or more processors to execute one or more of the processes. In some embodiments, the method 400 corresponds to the operation and training of the program synthesis module 130 (e.g., FIGS. 1-2) that performs program synthesis for a target problem. For example, as described with respect to FIGS. 1-2, the program synthesis module 130 may comprise one or more language models that are trained on one or more hardware processors (e.g., processor 101 of FIG. 1) by performing the same calculations on different hardware processors in parallel on different input data.

[0052] For example, TPU hardware with high-speed toroidal mesh interconnects can enable efficient parallel processing. To efficiently utilize the hardware, model training is implemented in JAX. Parallel evaluation in JAX employs the pjit() operator. This operator enables a paradigm called single-program, multiple-data (SPMD) code, which refers to a parallel processing technique in which the same computation is executed in parallel on different devices and on different input data. Specifically, pjit() is an API exposed to the XLA SPMD partitioner in JAX that allows a given function to be evaluated in parallel with equivalent semantics across a logical mesh of computations.

[0053] At step 402, a training data set may be received that includes pairs of text descriptions and corresponding code program snippets.

[0054] In step 404, the batch of training data may be partitioned into multiple subsets of training data.

[0055] In step 406, multiple subsets of the training data may be distributed across multiple virtual machines, each on one or more hardware processors. For example, the library JAXFORMER may recruit a designated coordinator node to orchestrate a cluster of TPU-VMs using a custom TCP / IP protocol. For data parallelism, the coordinator partitions the batches and partitions the partitions to individual TPU-VMs.

[0056] In one implementation, for model parallelism, two schemes for sharding of model parameters are supported: (1) Intra-TPU-VM, where parameters are sharded across MXU cores within a physical TPU-v4 board and replicated across the board; and (2) Inter-TPU-VM, where parameters are sharded across TPU-v4 boards and activations are replicated. Both the Intra-TPU-VM and Inter-TPU-VM sharding schemes are implemented based on a specific pjit() logical mesh specification (r, p, c), where r is the number of parameter replicas, p is the number of parameter partitions, and c is the number of n partitions, respectively. c n logic cores b is the number of logical cores per board across TPU boards, where d × p = n b and r×p×c=n b ×n c It is designed to be like this.

[0057] In step 408, the program synthesis module 130 may determine whether the total size of one or more language models is greater than a predefined threshold. For example, the intra-TPU-VM scheme may be employed for models whose size is 6B parameters or less, and the total amount of model and optimizer parameters fits into the combined HBM memory of a single TPU-v4 board.

[0058] In response to determining at step 410 that the total size of one or more language models is less than or equal to a predefined threshold, method 400 may proceed to step 412, where the total amount of model parameters is replicated across several TPU boards. For example, a TPU-v4-512 slice with nb=64 and nc=4 is configured as (r,p,c)=(64,1,4). That is, parameters are replicated across r=64 boards with p=1 total inter-board partitions and intra-board parallelism across c=4 logic chips. In this configuration, average gradients are accumulated across boards via with_sharding_constraint(), effectively emulating the behavior of the xmap() operator.

[0059] In response to determining in step 410 that the total size of one or more language models is greater than a predefined threshold, the method 400 may proceed to step 414, where the module shares the total amount of model parameters across several TPU boards. For example, an inter-TPU-VM scheme is adopted for models with a size of more than 6B parameters, where model and optimizer parameters need to be sharded across TPU-v4 boards. For example, a TPU-v4-512 slice with nb=64 and nc=4 is configured as (r,p,c)=(1,64,4). For larger slices, such as a TPU-v4-1024 with nb=128, redundancy may be introduced in the parameter sharding, e.g., (r,p,c)=(2,64,4). In this configuration, activations are replicated across boards via with_sharding_constraint(). Furthermore, (r,p,c) enables backward compatibility for logical hardware layout migration from a TPU-v3 with c=8 to a TPU-v4 with c=4 by adjusting p without requiring resharding.

[0060] Method 400 then proceeds to step 416, where the modules continue to train the language models in parallel on different hardware processors. For example, an Adam optimizer (described in Kingma et al., Adam: A method for stochastic optimization, in proceedings of International Conference on Learning Representation (Poster), 2015) may be employed for training with parameters (β1, β2, ∈) = (0.9, 0.999, 1e-08) and a global gradient norm clipping of 1.0. The learning rate function over time may follow GPT-3 (described in Brown et al., Language models are few-shot learners, Advances in neural information processing systems, 33:1877-1901, 2020) with a warm-up step and cosine annealing.

[0061] 5 is a simplified block diagram illustrating an example of program composition through multi-turn conversation, according to an embodiment described herein. The illustrative example 500 shows a multi-turn programming benchmark that performs the task of extracting a username from an email address. For example, a natural language description of the task can be decomposed into a series of prompts p1-p5 that are executed in multiple turns, as shown by the designed discourse 501.

[0062] In generation 502, each turn consists of a prompt and a generated response by one or more language models. Prompts p1-p5 can include templates that accept various test cases (i.e., {input} for different task descriptions). For example, for each problem, five test cases can be provided, each test case including (a) an input 505 that replaces the {input} in the prompt before the prompt is fed to the model, and (b) an expected output 510 against which the output of the generated program is compared to evaluate its functional correctness. In the example shown 500, the input 505 is a string containing abc.xyz@example.com, which replaces the {input} in p2, and the expected output is abc xyz. The model is conditioned on the concatenation of interleaved past prompts and generated responses.

[0063] In generation 502, at each term, the language model generates subprograms s1-s5 based on prompts p1-p5 and previously generated subprograms. For example, in turn 1, subprogram s1 is generated using prompt p1 as input. In turn 2, subprogram s2 is generated using the concatenated input of p1, s1, and p2. The process may continue until all five prompts p1-p5 have been processed. The resulting subprograms s1-s5 in each turn are then concatenated to form the final output code program 515.

[0064] During evaluation 503 , an output code program 515 may be executed to generate actual output results 520 , which are compared to the expected output 510 .

[0065] During training, the input training data may include a problem description and an expected output 510 for the problem. Thus, a training objective (e.g., cross-entropy) may be calculated by comparing the actual output 520 generated from the language model with the expected output 510. The training objective may then be used to update the language model. Data Experiment Example

[0066] In one embodiment, functional accuracy (as described in Hendrycks et al., Measuring coding challenge competence with APPS, in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021) is used to evaluate the model for program synthesis. Evaluation is performed on single-turn and multi-turn benchmarks. Both benchmarks are handwritten to avoid data leakage from the training data. We evaluate the model on both the single-turn evaluation benchmark and the proposed multi-turn benchmark.

[0067] HumanEval is a benchmark for measuring the program synthesis capabilities of models in Python (MIT license). It contains 164 handwritten programming problems. Each problem provides a prompt with a description of the function to be generated, the function signature, and an example test case in the form of an assertion. The model must complete the function given the prompt such that it can pass all the provided test cases. Since the user intent is specified in a single prompt and provided to the model once, an evaluation on HumanEval is considered a single-turn evaluation, distinct from a multi-turn evaluation, where the user intent is decomposed into multiple steps. We sample 158 nuclei with top-p (p=0.95) following the sampling strategy described in Holtzman et al., The curious case of neural text degeneration, in Proceedings of the 2020 International Conference on Learning Representation.

[0068] Model 130 is compared to the Codex model described in Chen et al., Evaluating large language models trained on code, arXiv preprint arXiv:2107.03374, 2021, which shows state-of-the-art performance in HumanEval. Additionally, program synthesis model 130 is compared to the open source large-scale language models GPT NEO (Black et al., GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021) and GPT-J (Wang et al., GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, May 2021). These are trained on THEPILE (described in Gao et al., The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020) and are therefore similar to our CODEGEN-NL models in terms of training data and model size. All models are evaluated at temperatures t ∈ {0.2, 0.6, 0.8} and pass@k is calculated for each model with k ∈ {1, 10, 100}. The temperature that gives the best-performing pass@k for each k is selected for direct comparison with the results of Chen et al., Evaluating large language models trained on code, arXiv preprint arXiv:2107.03374, 2021. The results of the program synthesis model 130 and the baselines are summarized in Figure 6.

[0069] As shown in Figure 6, the CODEGEN-NL model (350M, 2.7B, 6.1B) outperforms or is comparable to the respective GPT-NEO and GPT-J models. Further training of CODEGEN-NL on multilingual programming language data (BIGQUERY) results in CODEGEN-MULTI. The multilingual CODEGEN model significantly outperforms models trained on THEPILE (GPT-NEO, GPT-J, CODEGEN-NL). CODEGEN MULTI is then fine-tuned on a Python-only dataset (BIGPYTHON) to obtain CODEGEN-MONO. The program synthesis capability is significantly improved. Thus, Python program synthesis capability increases with increasing amounts of Python training data. For almost all models, as expected, increasing the size of the model improves overall performance. The Python-monolingual CODEGEN model performs competitively or with improved performance compared to the current state-of-the-art model, Codex. CODEGEN-MONO 2.7B performs worse than CODEX 2.5B when k = 100, but better when k ∈ {1, 10}. Although only half the size of CODEX 12B, CODEGEN-MONO 6.1B achieves a pass@k score close to that of the best performing Codex, CODEX 12B. Our CODEGEN-MONO 16.1B outperforms it.

[0070] The success of a program synthesis system depends heavily on how well it understands user intent. When the system is based on a language model, the perplexity of the problem prompts provides a proxy for the system's understanding of the user intent specification. A low perplexity of the intent specification under a model indicates that this intent specification is compatible with the knowledge learned by the model from the training data. Data experiments are then conducted to examine whether better prompt understanding, using lower prompt perplexity as a proxy, leads to more functionally accurate programs. All problems are partitioned into pass and non-pass. A pass problem is a problem where at least one sample out of the 200 samples passes all test cases, while a non-pass problem is a problem where none of the 200 samples passes all test cases. Based on the samples from the CODEGEN-MONO model, the average perplexity of the problem prompts for the pass problems is calculated, and the average perplexity of the problem prompts for the non-pass problems is calculated. The results are shown in Figure 7. The prompts for the pass problems have a lower perplexity than the prompts for the non-pass problems. This finding means that if the user intent specification is better understood by the model, the program synthesis is more likely to be successful. Indeed, some training data contains interleaved sequences of natural language comments and programs, where the comments describe the functions of the subsequent programs. Therefore, it is speculated that user intent specifications similar to such patterns will be better understood by the model and thus lead to better program synthesis. Therefore, the user intent may be specified in multiple turns so that the model can focus on a partial problem at a time, which makes it easier for the model to understand the user intent.

[0071] In some embodiments, it is a conversational program synthesis paradigm in which the process of synthesizing a program is cast as a multi-turn conversation between a user and a system, similar to the one illustrated in FIG. 5. To validate such a paradigm, a multi-turn programming benchmark (MTPB) is developed. The MTPB consists of 115 problems written by experts. The description to solve each problem is decomposed into multiple steps, and each step includes a description in natural language (prompts). To solve the problem, the model needs to synthesize a functionally correct subprogram (1) according to the description in the current step, and (2) taking into account the description in the previous step and the synthesized subprogram (e.g., correct back-references of functions and / or variables defined in the previous step).

[0072] In one embodiment, the MTPB may be constructed by curating a set of 115 problems that require a diverse range of programming knowledge, including problems requiring mathematical problems, array operations, string manipulation, algorithms, data science, and other knowledge. For each problem, a multi-turn prompt is included that serves as step-by-step instructions to the model, ensuring that (1) it forms at least three turns and (2) the problem solution does not belong to one turn. Five test cases are included for each problem to evaluate the synthetic program for functional correctness. In creating the test cases, non-trivial test cases are included that pass only if a correct solution is given. Unlike HumanEval, where the model is expected to complete a partially defined function, the MTPB problem only provides a prompt, so the model must generate a solution from scratch. While free-form generation may allow for a larger number of potential solutions, the lack of an entry point to provide test case inputs makes it difficult to test the generated code on a diverse set of test cases. To overcome this challenge, the test case inputs are embedded within the prompts. Specifically, prompts are written as Python formatted strings8, and the input values ​​are substituted for variable names when a particular test case is applied to a problem. For example, the prompt "Define a string named 's' with the value {var}." combined with a test case input var = 'Hello' will be formatted to "Define a string named 's' with the value 'Hello'."

[0073] In one embodiment, the history of pairs of prompts and generated completions are concatenated into a self-contained program for execution (see example shown in FIG. 5). The program is then executed in an isolated Python environment according to the single-turn HumanEval benchmark. However, HumanEval problems are constructed such that known function signatures are completed, so that invocation of the generated code under a set of functional unit tests is trivial. In the multi-turn case, such an entry point (or return value) is not guaranteed to be generated. To avoid the problem of missing return signatures (or return values), the last prompt of a multi-turn problem in MTPB is always specified to print out the resulting state to the terminal. The benchmark execution environment then overloads the Python print(args) function and stores the args on a stack. If the code sampled for the last prompt of the problem does not contain a print() statement, which is a valid convention for printing on the terminal in Python or specifically in Jupyter notebooks, the AST of the generated code is modified to insert a call to print(). Finally, a type-relaxed equivalence check (e.g., implicit conversions between lists and tuples) of the args is performed against the pre-defined gold outputs of the problem to determine the failure or success of the test.

[0074] In this analysis, data experiments are further conducted to investigate how model size and data size affect program synthesis ability in a multi-turn conversational paradigm. The language model is trained on the following datasets with four sizes: 350M, 2.7B, 6.1B, and 16.1B: THEPILE, BIGQUERY, and BIGPYTHON, which contain more and more Python data (see Section 2.1 for further details). The GPT-NEO, GPT-J, and CODEGEN-NL models are trained on THEPILE. The CODEGEN-MULTI model is initialized with the CODEGEN-NL model and then trained on BIGQUERY. The CODEGEN-MONO model is initialized with the CODEGEN-MULTI model and then trained on BIGPYTHON. In MTPB, each problem has five test cases, and 40 samples are sampled for each test case in each model, based on which the pass rate of each problem is calculated. The MTPB evaluation results (average pass rate) of the CODEGEN model and the baseline are shown in Figure 8. Clearly, performance on MTPB is demonstrated as a function of model size and data size, suggesting that the ability of conversational program synthesis scales as a function of model size and data size. Models are simply trained for autoregressive language modeling purposes. As models and data scale up, conversational capabilities emerge, i.e., the ability to synthesize programs in a multi-turn fashion.

[0075] Multi-turn decomposition improves the model's understanding of user intent specifications, which leads to improved program synthesis ability. To test this hypothesis, we made the multi-turn specifications single-turn counterparts by concatenating each specification into a single turn. We compare the perplexity of multi-turn prompts with that of concatenated single-turn prompts under the four CODEGEN-MONO models. The average perplexity across all problems in the MTPB is shown in the left panel of Figure 9. For all models, single-turn specifications have a higher average perplexity than multi-turn specifications. This means that multi-turn user specifications can be better understood by the model. The average perplexity for both multi-turn and single-turn intent specifications in the larger models is slightly lower than that in the smaller models, indicating that the larger models understand user intent better than the smaller ones.

[0076] Program synthesis is then compared for pass rates with multi-turn prompts versus concatenated single-turn prompts. The results are shown in the right panel of Figure 9. The multi-turn specifications are close to or more than 10 percentage points higher than the single-turn specifications for all model sizes. Combined with the perplexity analysis above, we believe that by decomposing user specifications into multiple steps and leveraging the emergent conversational capabilities of large language models, we can more easily digest user specifications and better synthesize programs.

[0077] Furthermore, problems can be classified by difficulty based on their average pass rate (less than 30% is “hard” and more than 70% is “easy”) to examine the interaction effect between difficulty and model size on the improvement due to multi-turn decomposition. The results are shown in Figure 10. Across almost all model sizes and difficulties, multi-turn prompts lead to significant improvements over single-turn prompts, with most improvements being near or above 10 percentage points. Interestingly, larger models (6.1B and 16.1B) are invariant to multi-turn decomposition for easy problems (see the two short bars 0.19% and −0.25% in Figure 10). This means that if a problem can be easily understood by a model (due to the combined effect of the ease of the problem and the higher power of the larger model), decomposition of the specification is unnecessary or unhelpful. This is indeed consistent with the motivating assumption of this application that decomposing a complex specification makes the problem easier to understand and improves program synthesis.

[0078] This description and the accompanying drawings illustrating aspects, embodiments, implementations, or applications of the present invention should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of the present specification and claims. In some instances, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure the embodiments of the present disclosure. Similar numbers in two or more figures represent the same or similar elements.

[0079] In this description, specific details are described that describe some embodiments consistent with the present disclosure. Numerous specific details are described to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative, not limiting. One skilled in the art may recognize other elements that are not specifically described herein but are within the scope and spirit of the present disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments, unless specifically described otherwise or unless one or more features render the embodiment non-functional.

[0080] Although illustrative embodiments have been shown and described, the foregoing disclosure contemplates a wide range of modifications, changes, and substitutions, and in some cases, some features of the embodiments may be employed without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention should be limited only by the scope of the claims that follow, which claims are appropriately interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. 1. A system for code program synthesis for a target problem, comprising: a communications interface configured to receive a natural language specification of a task problem; wherein the specification is broken down into a number of prompts; a memory storing one or more language models and a plurality of processor-executable instructions; one or more hardware processors; wherein the one or more hardware processors read the plurality of processor-executable instructions to: obtaining a first prompt from the plurality of prompts at a first generation time step; generating a first sub-program in response to the first prompt at the first generation time step with the one or more language models; obtaining a second prompt from the plurality of prompts at a second generation time step; generating, at the second generation time step, a second sub-program in response to the second prompt and based on the first sub-program and the first prompt with the one or more language models; concatenating generated subprograms from a series of generated time steps corresponding to the plurality of prompts into an output code program; generating a result of the task problem by executing the output code program; A system that performs operations including:

2. 2. The system of claim 1, wherein the specification is paired with an expected outcome of the task problem and the generated results are compared to the expected results to assess accuracy of the one or more language models.

3. 2. The system of claim 1, wherein the one or more language models comprise one or more autoregressive Transformer models trained with a next token prediction language modeling objective.

4. 10. The system of claim 1, wherein the one or more hardware processors comprise a tensor processing unit (TPU) having a high speed toroidal mesh interconnect.

5. The system of claim 4 , wherein the one or more language models are trained on the one or more hardware processors by performing the same calculations in parallel on different input data on different hardware processors.

6. The operation includes: Partitioning the batch of training data into a plurality of subsets of the training data; distributing a plurality of subsets of the training data to a plurality of virtual machines of the one or more hardware processors, respectively; The system of claim 5 further comprising:

7. The operation includes: replicating a total amount of model parameters across several TPU boards in response to determining that a total size of the one or more language models is less than or equal to a predefined threshold; in response to determining that a total size of the one or more language models is greater than the predefined threshold, sharing the total amount of model parameters across the number of TPU boards; The system of claim 6 further comprising:

8. 1. A method for code program synthesis for a target problem, comprising: receiving, via a communications interface, a natural language specification of a task problem, the specification being broken down into a plurality of prompts; obtaining a first prompt from the plurality of prompts at a first generation time step; generating a first subprogram in response to the first prompt at the first generation time step using one or more language models implemented on one or more hardware processors; obtaining a second prompt from the plurality of prompts at a second generation time step; generating, in the second generation time step, a second sub-program in response to the second prompt and based on the first sub-program and the first prompt, with the one or more language models; concatenating generated subprograms from a series of generated time steps corresponding to the plurality of prompts into an output code program; generating a result of the task problem by executing the output code program; The method includes:

9. 10. The method of claim 8, wherein the specification is paired with an expected outcome of the task problem and the generated results are compared to the expected results to assess accuracy of the one or more language models.

10. 10. The method of claim 8, wherein the one or more language models comprise one or more autoregressive Transformer models trained with a next token prediction language modeling objective.

11. The method of claim 8 , wherein the one or more language models are trained on the one or more hardware processors by performing the same calculations in parallel on different input data on different hardware processors.

12. Partitioning the batch of training data into a plurality of subsets of training data; distributing a plurality of subsets of the training data respectively across a plurality of virtual machines of the one or more hardware processors; The method of claim 11 further comprising:

13. In response to determining that a total size of the one or more language models is less than or equal to a predefined threshold, replicating a total amount of model parameters across a number of TPU boards; in response to determining that a total size of the one or more language models is greater than the predefined threshold, sharing the total amount of model parameters across the number of TPU boards; The method of claim 12 further comprising:

14. 1. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for code program synthesis for a target problem, the instructions comprising: receiving, via a communications interface, a natural language specification of a task problem, the specification being broken down into a plurality of prompts; obtaining a first prompt from the plurality of prompts at a first generation time step; generating a first subprogram in response to the first prompt at the first generation time step with one or more language models implemented on one or more hardware processors; obtaining a second prompt from the plurality of prompts at a second generation time step; generating, at the second generation time step, a second sub-program in response to the second prompt and based on the first sub-program and the first prompt with the one or more language models; concatenating generated subprograms from a series of generated time steps corresponding to the plurality of prompts into an output code program; generating a result of the task problem by executing the output code program; 2. A non-transitory processor-readable storage medium that is executed by the one or more hardware processors to perform operations that include:

15. 15. The non-transitory processor-readable storage medium of claim 14, wherein the specification is paired with an expected outcome of the task problem and the generated results are compared to the expected results to assess accuracy of the one or more language models.

16. 15. The non-transitory processor-readable storage medium of claim 14, wherein the one or more language models comprise one or more autoregressive Transformer models trained with a next token prediction language modeling objective.

17. 15. The non-transitory processor-readable storage medium of claim 14, wherein the one or more hardware processors comprise a tensor processing unit (TPU) having a high speed toroidal mesh interconnect.

18. The one or more language models are trained on the one or more hardware processors by performing the same calculations on different input data in parallel on different hardware processors, the operation comprising: Partitioning the batch of training data into a plurality of subsets of the training data; distributing a plurality of subsets of the training data to a plurality of virtual machines of the one or more hardware processors, respectively; 20. The non-transitory processor-readable storage medium of claim 17, further comprising:

19. The operation includes: replicating a total amount of model parameters across several TPU boards in response to determining that a total size of the one or more language models is less than or equal to a predefined threshold; in response to determining that a total size of the one or more language models is greater than the predefined threshold, sharing the total amount of model parameters across the number of TPU boards; 20. The non-transitory processor-readable storage medium of claim 18, further comprising:

20. The operation includes: generating a question for a user using the one or more language models; receiving a response from the user in response to the query; generating the second sub-program further based on the received response; 15. The non-transitory processor-readable storage medium of claim 14, further comprising:

Images