Controlling robots using language model generated programs

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling a robot using language model programs. A language model program is a computer program generated from an output of a code generation neural network, e.g., one that has been trained on a language modeling objective on computer code data.

Description

CONTROLLING ROBOTS USING LANGUAGE MODEL GENERATED PROGRAMSCROSS-REFERENCE TO RELATED APPLICATION[1] This application claims priority to U.S. Provisional Application No. 63 / 538,780, filed on September 15, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.BACKGROUND[2] This specification relates to processing data using machine learning models.[3] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.[4] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to a received input to generate an output.SUMMARY[5] This specification generally describes a system implemented as computer programs on one or more computers in one or more locations that controls a robot interacting with an environment using language model generated programs (LMPs). Particular implementations may be used to control a mechanical robot operating in a real-world environment.[6] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.[7] Providing users the ability to control robots using natural language instructions can be desirable for many types of robotics tasks.[8] However, implementing a policy that successfully causes a robot to follow an arbitrary user instruction is difficult. In particular, robots that use language need it to be grounded (or situated) to reference the physical world and bridge connections between words, perceived objects in the environment, and actions.[9] Some conventional methods ground language using lexical analysis to extract semantic representations that inform policies, but these techniques struggle to handle unseen instructions.

[0010] Some other methods attempt to learn the grounding end-to-end, but these techniques require copious amounts of training data, which can be expensive to obtain on real-world robots.

[0011] This specification, on the other hand, describes techniques for using a pre-trained code generation neural network, e.g., a language model pre- trained on computer code data, to generate robot policy code (“language model generated programs”) given natural language commands. In particular, the generated code can express functions or feedback loops that process perception outputs (e.g., outputs of open vocabulary object detectors) and parameterize control primitive Application Programming Interfaces (APIs) for controlling the robot. In particular, the LMPs can take in new natural language commands and autonomously re-compose the available API calls to generate new policy code that carries out the commands.

[0012] Thus, the described system can generalize to new instructions and can operate on arbitrary sets of available APIs, allowing the outputs of the system to move the robot in the environment as well as generate natural language text responses to questions posed in the natural language instructions.

[0013] More specifically, this specification describes how to generate an input sequence to the code generation neural network by including an instruction-based prompt in the input sequence. The instruction-based prompt includes natural language descriptions of the robot and the environment. This allows the code generation neural network to effectively incorporate information about the particular robot, environment, and task and to generate an accurate LMP for controlling the particular robot given a natural language instruction of what task to perform.

[0014] For example, another approach is to rely on examples of robot code for existing tasks in the prompt to cause the code generation neural network to generate code for new tasks, and assume that the code generation neural network can infer task information, constraints, and API usage from examples alone. But high-quality examples can be costly to generate, e.g., because they require validating generated code on a real-world, physical robot, and too few or too many examples can bias the code generation neural network in the wrong direction. By using natural language instruction-based prompting, e.g., formatted as commented code, the system allows the code generation neural network to generalize to different robots, tasks, and sets of available APIs.

[0015] Moreover, this specification also describes how to combine these instruction-based prompts with other types of information in the input to the code generation neural network to further boost the performance of the code generation neural network in controlling robots. For example, for some tasks, documenting the task and providing full robot API descriptions can present a challenge to context-length limits of the code generation neural network. That is, documenting the task and providing a full API description of all available APIs can exceed the maximum context length for the neural network.

[0016] To account for this, the system can combine the instruction-based prompt with additional types of information, e.g., few-shot examples, chain of thought (CoT) prompting, or both, to generate code for planning a variety of tasks with higher success rate than previous works.

[0017] Additionally, in some cases, the code generation neural network may struggle to generate LMPs that specify a feasible policy that can actually be followed by the robot in the environment. To assist with this, the described techniques can include information in the prompt that causes the code generation neural network to, as part of generating an LMP, generate predictions of the state of the environment, i.e., as a consequence of actions specified in the LMP. Including these state predictions can result in more feasible, grounded LMPs, and improve the control of the robot.

[0018] As a particular example, by making use of the described prompting techniques, the code generation neural network can generate LMPs that define novel motion primitives on the fly just by changing the input task instruction. As one example of this, for a mobile manipulator robot, by using a prompt with a description of the robot, its constraints, robot APIs, as well as an example of robot code for grasping an object on a countertop, the code generation neural network can be queried to generate new motion primitives for multiple tasks. The generated code exhibits “motion commonsense” knowledge from the training of the neural network which are required to solve these low-level control tasks, including understanding of how a gripper should be oriented with respect to objects (e.g., vertically for a drawer handle, or horizontally for a knob).

[0019] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG. 1 shows an example robot policy system.

[0021] FIG. 2 is a flow diagram of an example process for controlling the robot.

[0022] FIG. 3 is a flow diagram of an example process for generating a computer program from an output sequence.

[0023] FIG. 4 shows an example of an LMP.

[0024] FIG. 5 shows an example of a portion of an input sequence processed by the code generation neural network.

[0025] FIG. 6 is a flow diagram of an example process for refining an instruction-based prompt.

[0026] FIG. 7 shows an example of the performance of the described techniques.

[0027] FIG. 8 shows another example of the performance of the described techniques.

[0028] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0029] FIG. 1 shows an example robot policy system 100. The robot policy system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0030] The robot policy system 100 controls a robot 104 interacting with an environment 106 to accomplish tasks.

[0031] The robot 104 may be a mechanical robot operating in a real- world environment. The robot 104 can be any appropriate type of robot, e.g., e.g., a robotic arm, a humanoid robot, a quadruped robot, a vehicular robot, e.g., an autonomous vehicle, and so on.

[0032] As a general example, the task can include one or more of, e.g., navigating to a specified location in the environment, identifying a specific object in the environment, manipulating the specific object in a specified way, and so on. For example, tasks can include identifying and sorting diverse objects in the environment.

[0033] In some cases, and as will be described below, the task is specified by a natural language instruction 122 that is received by the system 100.

[0034] The robot 104 generally has one or more sensors 105 that sense the environment, e.g., one or more of the following: camera sensors, laser sensors, radar sensors, temperature sensors,microphone sensors, proprioceptive sensors, e.g., optical encoders, gyroscopes, gyroscopes, accelerometers, inertial measurement units, and so on, etc.

[0035] The robot 104 generally also has software that processes readings from these sensors to generate outputs, e.g., machine learning models or other software that detects, localizes, or otherwise characterizes objects in the environment. For example, the robot 104 can have machine learning models that perform object localization, that perform open-vocabulary object detection, and so on.

[0036] The robot 104 is controlled by specifying parameters 108 for one or more control primitives. A control primitive is a function that maps a set of parameters to a sequence of one or more control inputs to the robot. For example, a control primitive can be a low-level control primitive that maps a trajectory, e.g., a sequence of waypoint locations for the robot, to a sequence of control inputs that cause the robot to follow the trajectory. Each control input may control an actuator of the robot to move the robot.

[0037] A control system 107 for the robot 104 then maps these control primitives 108 into control inputs for one or more controllable elements of the robot, e.g., position, velocity, or force / torque / acceleration data for one or more joints or other control elements of the robot. That is, the control system 107 performs the function specified by the control primitive 108 to map the set of parameters to the corresponding sequence of control inputs.

[0038] More specifically, the system 100 controls the robot through language model generated programs (LMPs) 122.

[0039] An LMP 122 is a computer program generated from a computer code sequence generated as output by a code generation neural network 120.

[0040] A code generation neural network 120 is a neural network that receives an input sequence of tokens in a vocabulary for a programming language, e.g., Python, C++, C, or another programming language, and processes the input sequence to generate an output sequence of tokens in the vocabulary that specifies a computer program in the programming language.

[0041] For example, the code generation neural network 120 can have been trained on a language modeling objective on a corpus of training computer code sequences from training computer programs. That is, the code generation neural network 120 can have been trained on a next token prediction task that requires the neural network 120 to predict the next token in a computer code sequence given the preceding tokens in the computer code sequence, a maskedlanguage modeling task that requires the neural network 120 to predict missing (or “masked”) tokens in computer code sequences given the other tokens in the computer code sequence, or both.

[0042] Generally, the corpus of training data used for training the code generation neural network 120 can include other types of data, e.g., natural language data, structured data, and so on.

[0043] In some cases, the code generation neural network 120 can have been further trained on one or more additional objectives after having been pre-trained on an initial objective, e.g., the language modeling objective described above. For example, the additional objective(s) can include one or more of instruction-tuning, reinforcement learning from human feedback, supervised fine-tuning, and so on.

[0044] As a particular example, the code generation neural network 120 can have the architecture of an auto-regressive language model neural network.

[0045] The language model neural network is referred to as an auto-regressive neural network because the neural network auto-regressively generates an output sequence of tokens by generating each particular token in the output sequence conditioned on a current input sequence that includes any tokens that precede the particular text token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token, and a context input that provides context for the output sequence. For example, the current input sequence when generating a token at any given position in the output sequence can include the context sequence and the tokens at any preceding positions that precede the given position in the output sequence. As a particular example, the current input sequence can include the context sequence followed by the tokens at any preceding positions that precede the given position in the output sequence. Optionally, the context and the current output sequence can be separated by one or more predetermined tokens within the current input sequence.

[0046] More specifically, to generate a particular token at a particular position within a candidate output sequence, the neural network 120 can process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each token in the vocabulary of tokens. The neural network 120 can then select, as the particular token, a token from the vocabulary using the score distribution. Forexample, the neural network 120 can greedily select the highest-scoring token or can sample, e.g., using nucleus sampling or another sampling technique, a token from the distribution.

[0047] As a particular example, the language model neural network 120 can be an autoregressive Transformer-based neural network that includes (i) a plurality of attention blocks that each apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution.

[0048] The neural network 120 can have any of a variety of Transformer- based neural network architectures. Examples of such architectures include those described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute- optimal large language models, arXiv preprint arXiv:2203.15556, 2022; J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer,R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Eisen,S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d’Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a humanlike open-domain chatbot. CoRR, abs / 2001.09977, 2020; Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprintarXiv:2005.14165, 2020; and Anil, et al, PaLM 2 Technical Report, arXiv preprint arXiv:2305.10403.

[0049] Generally, however, the Transformer-based neural network includes a sequence of attention blocks, and, during the processing of a given input sequence, each attention block in the sequence receives a respective input hidden state for each input token in the given input sequence. The attention block then updates at least the hidden state for the last token in the given input sequence at least in part by applying self-attention to generate a respective output hidden state for the last token. The input hidden states for the first attention block are embeddings of the input tokens in the input sequence and the input hidden states for each subsequent attention block are the output hidden states generated by the preceding attention block.

[0050] In this example, the output subnetwork processes the output hidden state generated by the last attention block in the sequence for the last input token in the input sequence to generate the score distribution.

[0051] Generally, because the neural network 120 is auto-regressive, the system 100 can use the same neural network 110 to generate multiple different candidate output sequences in response to the same request, e.g., by using beam search decoding from score distributions generated by the neural network 110, using a Sample-and-Rank decoding strategy, by using different random seeds for the pseudo-random number generator that is used in sampling for different runs through the neural network 120 or using another decoding strategy that leverages the auto-regressive nature of the neural network.

[0052] The system 100 generally has access to a set of application programming interfaces (APIs). The APIs available to the system 100 can include one or more APIs for querying specific types of information obtained from sensor readings and one or more APIs for parameterizing the control primitives that are used to control the robot.

[0053] To control the robot 104, the system 100 obtains a natural language instruction 112 for interacting with the robot 104.

[0054] For example, the system 100 can receive the instruction 112 as input, e.g., as text or audio input, from a user that desires to cause the robot 104 to navigate through the environment, manipulate one or more objects in the environment, or to obtain information generated as a result of navigation of the robot through the environment.

[0055] For example, the instruction 112 can be an instruction for the robot 104 to move an object to a specified location in the environment, to navigate to a specified location in the environment, to locate a specified object in the environment, or to respond to a query about the environment.

[0056] The system 100 generates an input sequence 114 that includes (i) the natural language instruction 112, i.e., represented as tokens from the vocabulary, and (ii) an instruct! on- based prompt 116 that includes a natural language description of the robot and the environment.

[0057] That, is, the instruction-based prompt 116 provides natural language information about the environment, the task, and the robot to augment the natural language instruction 112.

[0058] For example, the natural language instruction 112 and the instruction-based prompt 116 can be formatted as comments within the input sequence 114.

[0059] The input sequence 114 will be described in more detail below with reference to FIG. 2.

[0060] The system 100 processes the input sequence 114 using the code generation neural network 120 to generate as output a computer code sequence in the programming language that defines a computer program for interacting with the robot.

[0061] Generally, the computer code sequence includes computer code representing a call to one or more of the APIs and arguments for the call. The one or more APIs include one or more APIs that can be called to control the robot, and, optionally, one or more APIs that can be called to obtain outputs generated by processing readings from the one or more sensors of the robot. That is, the APIs can include one or more APIs for parameterizing the control primitives that are used to control the robot and, optionally, one or more APIs for querying specific types of information obtained from sensor readings.

[0062] The system 100 generates, from the computer code sequence, the computer program, i.e., the LMP 122, and then executes the computer program to interact with the robot. That is, when the computer code sequence includes an API call to an API that parameterizes the control primitives, executing the computer program causes the robot to move in the environment. When the computer code sequence includes an API call to an API that provides information derived from sensor readings, executing the computer program causes the information to be retrieved and processed according to the remainder of the sequence.

[0063] In some cases, e.g., when the instruction 112 includes a question about the environment, the LMP 122 can return a natural language response (e.g. based on sensor readingsand / or information derived from sensor readings). The system 100 can then provide the response to the user that submitted the instruction 112, e.g., as natural language text or as speech using a text-to-speech engine.

[0064] The control inputs generated by the control system 106 to control the robot can include data for controlling one or more components (e.g. motors, sensors, etc.) of the robot. For instance, control inputs can include, e.g., torques for the joints of the robot or higher-level control commands. In other words, the control inputs can include for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or parts of another mechanical agent. Control inputs may additionally or alternatively include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment.

[0065] FIG. 2 is a flow diagram of an example process 200 for controlling the robot. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a robot policy system, e.g., the robot policy system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200.

[0066] The system obtains a natural language instruction for interacting with the robot (step 202).

[0067] The system generates an input sequence that includes (i) the natural language instruction, and (ii) an instruction-based prompt (step 204). As described above, both (i) and (ii) can be represented within the input sequence as commented code. Commented code may include computer executable code and one or more comments (e.g. natural language text / annotation(s)). The one or more comments may be marked (e.g. by one or more tags or delimiters) so as to be ignored (e.g. ignored by a compiler or interpreter) when executing the commented code.

[0068] The instruction-based prompt includes a natural language description of the robot and of the environment. This provides the code generation neural network information about the context in which the natural language instruction is to be followed.

[0069] The natural language description of the robot can include any of a variety of information characterizing the robot.

[0070] For example, the description can include a description of the state space of the robot. The state space of the robot indicates what information is available from the sensors of the robot about the current state of the robot or of the environment. For example, the description of the state space can include, for example, how poses, e.g., of components of the robot or of objects in the environment that are obtained by calling corresponding APIs are to be interpreted. As one example of this, for a robot API that uses 6D poses, the description can specify the pose formats and canonical directions as follows:# Pose is a diet with ’position’ and ’orientation’ keys in robot frame.# The ’position’ value is a 3D array of [x, y, z] coordinates in meters.# The ’orientation’ value is a 4D array quaternion in [w, x, y, z],# In the robot frame, directions are defined as follows:# Positive x / Negative x: Front and Back# Positive y / Negative y: Left / Right# Positive z / Negative z: Upward / Downward

[0071] As another example, the description can include a description of the action space of the robot. For example, the description can identify each of the degrees of freedom of the robot and can specify how each of those degrees of freedom is controlled.

[0072] As yet another example, the description can include a description of the embodiment of the robot e.g. can describe or indicate one or more characteristics, such as physical characteristics, of the robot. For instance, the description of the embodiment of the robot can describe the physical type of the robot, e.g., stationary robot, mobile robot, single arm robot, bimanual robot, quadrupedal robot, and so on. For example, for a stationary robot with two arms, the embodiment can be described as:# You are a stationary robot with two arms.

[0073] As yet another example, the description can include a description of one or more constraints of the robot. Constraints can include any of a variety of constraints that are imposed on the robot, e.g., constraints imposed by the user of the system, physical constraints that are not to be violated to avoid damage or wear and tear to the robot, constraints that identify infeasible actions or states, and so on. For example, an example set of constraints on a bimanual robot can be expressed as:# Constraints:# 1. The right arm can pick and place objects on the right bin and the table only. It cannot reach the left bin to pick or place objects.# 2. The left arm can place objects on the left bin and the table. It cannot reach the right bin to pick or place objects.# 3. You can handover objects from one arm to the other. You cannot do a handover if you don’t have an object in the passing arm or the receiving arm is not empty.# 4. You can only pick one object at a time per arm.

[0074] Similarly, the description of the environment can include any of a variety of information characterizing the environment.

[0075] As one example, the description can include a description of one or more objects in the environment or one or more other properties in the environment that may be relevant to following the natural language instruction. For example, the description of the environment can be combined with the description of the embodiment of the robot as follows:# You are a stationary robot with a left arm and right arm placed on a table with a bin to the right and another bin to the left.

[0076] As another example, the description can include a description of one or more environment constraints. Environment constraints can include constraints on objects in the environment or other aspects of the environment, e.g., imposed by a user or that indicate infeasible states for the objects or other aspects of the environment.

[0077] The instruction-based prompt can also include additional information, i.e., in addition to the natural language description of the robot and of the environment.

[0078] For example, the instruction-based prompt can include a description of the overall type of task being performed by the robot. For example, even when different instances of a sorting task include different objects that need to be sorted as specified by the natural language instruction, the prompt can include the following overall description:# Your task is to write code that will sort objects into the correct bins as defined from given instructions.

[0079] As another example, the instruction-based prompt can include additional instructions about how to incorporate the training of the code generation neural network into generating the LMPs. For example, the prompt can include the following instruction:# Use your best world knowledge and make any necessary assumptions when selecting objects to sort

[0080] As another example, the instruction-based prompt can include guidelines or requirements for the LMPs that are generated. For example, the prompt can include the following instruction:# Your response should be exclusively in the form of Python code and Python- formatted comments. Do not use additional if statements or loops. Only write code that calls the provided robot API.

[0081] As another example, the instruction-based prompt can include a Ushot prompt, where k is an integer greater than or equal to one. In other words, the input sequence further includes k, i.e., one or more, example sequences that each include (i) an example natural language instruction, and (ii) an example computer code sequence in the programming language generated in response to the example natural language instruction. Generally, the example computer code sequences will include calls to the APIs to control the robot and calls to the APIs to obtain outputs from the one or more sensors of the robot. By including the Ushot prompt in the input sequence, the system provides the code generation neural network information about what APIs are available to be called and examples of how the APIs can be used to carry out the natural language input. As described above, the example natural language instruction in each example can be formatted as a code comment while the computer code sequences are not.

[0082] One simplified example of a one-shot prompt can be as follows:# if you see an orange, move backwards. if detect_object(" orange"): robot. set_velocity(x=-0.1, y=0, z=0) where “# if you see an orange, move backwards” is an example natural language instruction formatted as commented code, “detect_object(“ orange”) is a call to an API that performs openvocabulary object detection in images captured by sensors of the robot, and“robot. set_velocity(x=-0.1 , y=0, z=0)” is a call to a control primitive API that sets the angular velocity of the robot to the parameters specified in the call, i.e., with the parameters being the x velocity -.1, the y velocity 0, and the z velocity 0.

[0083] In the simplified example above, the natural language instruction obtained by the system could for example include “move rightwards until you see the apple”. The API callspecified in the context sequence may then be re-composed by the pre-trained code generation neural network to generate a computer code sequence as follows: while not detect_object(“ apple”): robot. set_velocity(x=0, y=0.1, z=0)

[0084] In some implementations, the system incorporates Chain of Thought into one or more of the k examples. Chain of Thought (CoT) is a prompting method that writes the step-by-step reasoning process of solving a task in the prompt. The system can incorporate CoT in code generation by formatting each reasoning step as a comment in the example code sequence. Including CoT reasoning steps as commented code within the examples then causes the code generation neural network to generate reasoning steps as commented code as part of generating the LMP. This can improve the planning ability of the code generation neural network, improving the quality of the generated LMPs.

[0085] In some implementations, the system incorporates state predictions into one or more of the k examples. In particular, the system includes, in one or more of the example code sequences, example predictions of the state of the robot or of the environment, e.g., as obtained from one or more of the APIs, as a consequence of the robot taking the action or actions that precede the state prediction within the example code sequence. The system can format each state prediction as a comment in the example code sequence to prevent the state prediction from impacting the execution of the LMP. Including state predictions as commented code within the examples then causes the code generation neural network to generate state predictions as commented code as part of generating the LMP. This can improve the ability of the code generation neural network to plan and to generate feasible LMPs that obey the given constraints.

[0086] As another example, the prompt can include text documenting the one or more APIs. The documenting the one or more APIs can include text that identifies each of the APIs and provides a natural language description of the function of each of the APIs, e.g., also formatted as a code comment. The text documenting the one or more APIs can also include code defining each of the APIs. As one example, the prompt can include the following when the set of APIs includes one API for detecting objects, one API for a handover control primitive and another API for an object picking control primitive:Class Robot: def detect_objects(self, obj names):def handover(self, sour ce arm name, target arm name): def pick(self, arm name, obj name):

[0087] As another example, the prompt can include a description of the current state of the environment. As a particular example, the description of the current state of the environment can be formatted as computer code comments in the instruction-based prompt. The description of the current state of the environment can generally include descriptions of the current state of any appropriate aspect of the environment. As one example, the description can include a description of a corresponding positions of each of one or more objects in the environment. Alternatively, or in addition, the description can include a description of a respective state of each of one or more components of the robot.

[0088] For example, the system can obtain these descriptions by calling one or more of the available APIs. For example, to obtain the current position of a given object in the environment, the system can call an API that performs object detection with a query that specifies the given object.

[0089] When the examples make use of state predictions, including the description of the current state of the environment can cause the code generation neural network to include predicted states in the output sequence and, more specifically, can allow the code generation to make more accurate state predictions, improving the quality of the remainder of the output sequence.

[0090] As another example, instead of or in addition to the above, the prompt can include code that is formatted as import statements that inform the neural network which APIs are available and type hints, e.g., instructions or guides, on how to use those APIs.

[0091] Generally, the input sequence can include any combination of the above examples, optionally in combination with other data. For example, the input sequence can include tokens representing sensor data, e.g., image, video, Lidar, or proprioceptive sensor data, in addition to or instead of the natural language description of the robot or of the environment or both.

[0092] In some implementations, the natural language instruction can be a follow-up natural language instruction that follows a preceding instruction (and corresponding movement / control of the robot). In these implementations, the input sequence can also include the preceding natural language instruction and the output sequence generated for the preceding natural language instruction. This can allow the code generation neural network to generate programsthat use context from previously generated programs and allow users to submit feedback that results in modification to the control of the robot.

[0093] The system processes the input sequence using a code generation neural network to generate as output a computer code sequence in a programming language that defines a computer program that represents a policy for controlling the robot (step 206). Generally, the computer code sequence includes computer code representing a call to one of the one or more APIs and arguments for the call.

[0094] In some cases, because the code generation neural network has been pre-trained on a code prediction task as described above, the computer code sequence can also include code that represents a call to a library for the programming language that is not specified in the context sequence, e.g., by using NumPy or other similar libraries to elicit spatial reasoning with coordinates.

[0095] The system generates, from the computer code sequence, the computer program (step 208) and executes the computer program to control the robot (step 210).

[0096] In this way, the system may control the robot in accordance with the computer program.

[0097] To execute the computer program, the system can call an execution function for the programming language with an input string that is derived from the computer code sequence, e.g., with an input string that formats the computer code sequence according to a syntax for the execution function.

[0098] More specifically, the system can call the execution function with the input string and with (i) a global dictionary that specifies the one or more APIs and (ii) a local dictionary that can be populated with variables defined during the execution.

[0099] The local dictionary can be initialized as empty when the execution function is called and can be populated with values as the computer program is executed. For some natural language instructions, after the computer program has been executed, the system can provide one or more values from the local dictionary in response to the natural language instruction.

[0100] For example, the natural language instruction may include a request for the system to provide information about the environment to the user, e.g., from whom the natural language instruction was received. As part of executing the program, the system populates the local dictionary with the values that are specified in the request for information and, after execution,the system can provide the one or more values in response to the request. For example, the generated program can include a call to an API that outputs text and that parameterizes the call with one or more values stored in the local dictionary.

[0101] In some implementations, prior to executing the computer program, the system can first check to ensure that the computer program is safe to run. For example, the system can verify that there are no import statements, calls to certain functions that have been deemed risky, e.g., the “eval” or “exec” function in Python, or no special variables, e.g., variables that begin with “ ” in a Python program.

[0102] Additionally, in some implementations, the system generates computer programs that are hierarchical.

[0103] This is described in more detail below with reference to FIG. 3.

[0104] FIG. 3 is a flow diagram of an example process 300 for generating a computer program from an output sequence. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a robot policy system, e.g., the robot policy system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0105] The system can continue performing the process 300 until the computer code sequence does not include any functions that are undefined. Once the computer code sequence no longer includes undefined functions, the system can execute the computer program defined by the code sequences and the additional function definitions as described above.

[0106] The system determines whether the computer code sequence includes any computer code that calls a function that is undefined (step 302).

[0107] That is, the system identifies all of the function calls in the computer code sequence and then determines whether any of the function calls call a function that has not been defined, e.g., in a library for the programming language, within the computer code sequence, or within the local dictionary.

[0108] For example, in Python, the system can parse a code block’s abstract syntax tree to check for functions that do not exist in the given scope.

[0109] When the system identifies a function call that calls an undefined function, the system generates a new input sequence that includes a natural language instruction to define the function (step 304).[HO] For example, the new input sequence can include a natural language instruction “# define function: get_obj_bbox_area(obj_name),” where “get_obj_bbox_area(obj_name)” is the name of the function that was called in the computer code sequence and was determined to be undefined.[Hl] In some implementations, the new input sequence also includes one or more of the natural language instruction, the context sequence, or the output sequence generated by the code generation neural network. This can provide context to the code generation neural network for generating the definition of the undefined function.

[0112] The system then processes the new input sequence using the code generation neural network to generate a new output sequence that includes new computer code that defines the function (step 306).

[0113] The system then adds the new computer code that defines the function to the scope of the program. For example, the system can add data specifying the new computer code that defines the function to the global dictionary, the local dictionary, or both. As another example, the system can generate a modified computer code sequence that replaces the first computer code with the new computer code.

[0114] The system can then repeat the process 300 on the code that defines the function, e.g., to ensure that the function definition does not itself call any undefined function and, if not, that the code generation neural network is used to define the function. The system can continue performing iterations of the process 300 until all of the functions in the output sequence and all of the functions in the new output sequences have definitions.

[0115] Thus, in this manner, the system can implement hierarchical program generation, which allows the code generation neural network to hierarchically generate complex programs that when executed successfully cause the robot to respond to the natural language instruction. For example, the programs can sequence control primitives or build action trajectories.

[0116] FIG. 4 shows an example 400 of an LMP 410 generated by the code generation neural network in response to a natural language instruction 402.

[0117] In the example of FIG. 4, the instruction 402 is “Stack the blocks on the empty bowl.”

[0118] As shown in FIG. 4, the system processes an input sequence that includes the instruction 402 using the code generation neural network 120 to generate the LMP 410 written in Python.

[0119] As can be seen from the example, the LMP 410 calls several APIs. For example, the LMP 410 includes two API calls to an API that provides perception outputs (“detect objects”) and an API call to an API that parameterizes control primitives for the robot (“pick_place”).

[0120] Additionally, in the example of FIG. 4, the code generation neural network 120 has generated the LMP 410 hierarchically. In particular, an initial sequence 412 generated by the neural network 120 includes calls to two undefined functions: “is_empty” and “stack_objects.” In response to determining that these functions are undefined, the system has used the neural network 120 to generate new sequences 414 and 416 that define each of the functions. As can be seen from FIG. 4, the new sequences 416 that defines the “stack objects” function includes a call to the API that parameterizes control primitives. Thus, the system has hierarchically generated a complex string of control inputs for the robot that causes the robot stack multiple blocks on the empty bowl.

[0121] FIG. 5 shows an example 500 of a portion of an input sequence processed by the code generation neural network.

[0122] As can be seen from the example 500, the input sequence includes instructions 502 that specify the embodiment of the robot and the robot constraints. The input sequence also includes chain- of-thought reasoning steps 504 represented as commented code to enable the code generation neural network to perform multi-step reasoning when generating an LMP. Additionally, the input sequence also includes state predictions 506 also represented as commented code, which increases the likelihood that the LMP will represent a policy that is feasible for use in controlling the robot.

[0123] In some implementations, the system refines an initial instruction-based prompt, e.g., received from a user or generated by the system, in order to generate a final instruction-based prompt that results in the code generation neural network generating an improved LMP for controlling the robot.

[0124] An example of this is described below with reference to FIG. 6.

[0125] FIG. 6 is a flow diagram of an example process 600 for refining an instruction-based prompt. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a robot policy system, e.g., the robot policy system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0126] The system receives an initial instruction-based prompt for a new robot task domain (step 602). For example, the system can receive the initial prompt from a user, e.g., a robot engineer, or can automatically generate the initial prompt from information available about the new robot task domain. This initial prompt may be suboptimal, e.g., due to insufficient domain information, or style and presentation that is difficult for the code generation neural network to understand.

[0127] The system then performs a set of validation tasks using the initial prompt (step 604). In this stage, for a given validation task, the system receives a natural language instruction for an instance of the task, and then processes an input sequence that includes the natural language instruction and the initial prompt using the code generation neural network to generate an LMP. The system then determines whether the LMP fails. For example, the LMP can fail when the robot does not successfully perform the task as a result of the LMP being executed, e.g., when the LMP fails to execute, when the LMP specifies an infeasible policy that cannot be carried out by the robot, when the LMP specifies a policy that is feasible but does not effectively control the robot, or when the LMP specifies a policy that is feasible but that, when executed, does not control the robot to successfully perform the task (e.g. as determined based on one or more sensor readings representing one or more states of the environment during and / or at the end of the execution of the LMP).

[0128] If the LMP fails, the system provides error feedback to the code generation neural network that causes the code generation neural network to generate an improved LMP. The system repeats this process until the task is solved.

[0129] At the end of each validation task, the system obtains a sequence of (task, code, feedback) tuples. The feedback may specify whether the task has succeeded or failed.

[0130] The system uses the code generation neural network to generate an improved prompt given the tuples generated for the validation tasks (step 606). For instance, the system may generate an improved prompt in response to at least one of the validation tasks failing. The method 600 may be repeated until every validation task is completed successfully.

[0131] In particular, the system provides the code generation neural network with the initial prompt, the history of (task, code, feedback) tuples, and a request to improve the prompt, e.g., “How would you modify the initial prompt to avoid making this mistake in the future while keeping existing constraints?", “How would you add a general constraint to avoid making thismistake in the future?", or both. This causes the code generation neural network to generate modifications to the initial prompt. The system then applies these modifications to the initial prompt to generate a refined prompt that will cause the code generation neural network to perform better on subsequent tasks.

[0132] FIG. 7 shows an example 700 of the performance of the described techniques. As can be seen from FIG. 7, including the instruction-based prompt along with the few-shot examples improves the performance of two different robot embodiments across three different models, i.e., three different pre-trained code generation neural networks.

[0133] FIG. 8 shows another example 800 of the performance of the described techniques with and without state predictions. As can be seen from FIG. 8, including the state predictions in the input sequence significantly improves the performance of the code generation neural network on a trash sorting task, particularly when the input sequence includes the instruction-based prompt along with the few-shot examples.

[0134] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0135] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non- transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine- readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, orelectromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0136] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0137] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0138] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0139] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also beperformed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0140] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0141] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0142] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other formsof message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0143] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and computeintensive parts of machine learning training or production, i.e., inference, workloads.

[0144] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework.

[0145] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0146] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0147] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what can be claimed, but rather as descriptions of features that can be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially be claimed assuch, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0148] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0149] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing can be advantageous.

[0150] Aspects of the present disclosure may be as set out in the following clauses:Clause 1. A method for controlling a robot, the method comprising: obtaining a natural language instruction for interacting with the robot; generating an input sequence that comprises:(i) the natural language instruction; and(ii) an instruction-based prompt that comprises a natural language description of the robot and of the environment; processing the input sequence using a code generation neural network to generate as output a computer code sequence in a programming language that defines a computer program that represents a policy for controlling the robot; generating, from the computer code sequence, the computer program; and executing the computer program to control the robot.Clause 2. The method of clause 1 , wherein the computer code sequence comprises a call to one or more application programming interfaces (APIs), and wherein one or more of the APIs parameterize low-level control primitives for controlling the robot.Clause 3. The method of clause 2, wherein the one or more APIS that parameterize low- level control primitives for controlling the robot comprise a first API that parameterizes a first low-level control primitive that maps a trajectory of waypoint locations for the robot to a sequence of control inputs for the robot that cause the robot to follow the trajectory of waypoint locations.Clause 4. The method of clause 2 or clause 3, wherein one or more of the APIs can be called to obtain outputs from one or more sensors of the robot.Clause 5. The method of any preceding clause, wherein the input sequence comprises (iii) a ir-shot prompt that comprises k examples, wherein k is greater than or equal to one, and wherein each example comprises an example natural language instruction and an example computer program that, when executed to interact with the robot, causes the robot carry out the example natural language instruction.Clause 6. The method of clause 5 when dependent on clause 2, wherein the example computer programs in the k examples include calls to the one or more APIs.Clause 7. The method of any one of clauses 2-6, when dependent on clause 2 wherein the instruction-based prompt comprises text documenting the one or more APIs.Clause 8. The method of any preceding clause, wherein executing the computer program to interact with the robot comprises: calling an execution function for the programming language with an input string derived from the computer code sequence.Clause 9. The method of clause 8, when dependent on clause 2, wherein calling the execution function further comprises: calling the execution function with:(i) a global dictionary that specifies at least the one or more APIs, and(ii) a local dictionary that can be populated with at least variables defined during the execution.Clause 10. The method of clause 9, the method further comprising: after executing the computer program, providing one or more values from the local dictionary in response to the natural language instruction.Clause 11. The method of any preceding clause, wherein the code generation neural network is an auto-regressive neural network that auto-regressively generates each computed code token in the computer code sequence conditioned on the input sequence.Clause 12. The method of clause 11, wherein the code generation neural network is a Transformer neural network.Clause 13. The method of any preceding clause, wherein the natural language instruction specifies an instance of a task to be performed by the robot and wherein the instruction-based prompt comprises a natural description of one or more of: an objective of the task, constraints on performing the task, or a context for the task.Clause 14. The method of any preceding clause, wherein the natural language description of the robot and of the environment comprises: natural language description of a state space of the robot.Clause 15. The method of any preceding clause, wherein the natural language description of the robot and of the environment comprises: natural language description of an action space of the robot.Clause 16. The method of any preceding clause, wherein the natural language description of the robot and of the environment comprises: natural language description of an embodiment of the robot.Clause 17. The method of any preceding clause, wherein the natural language description of the robot and of the environment comprises: natural language description of one or more robot constraints.Clause 18. The method of any preceding clause, wherein the natural language description of the robot and of the environment comprises: natural language description of one or more environment constraints.Clause 19. The method of any preceding clause, wherein the instruction-based prompt further comprises: description of a current state of the environment.Clause 20. The method of clause 19, wherein the description of the current state of the environment is formatted as computer code comments in the instruction-based prompt.Clause 21. The method of clause 19 or 20, wherein the description of the current state of the environment comprises one or more of: description of positions of one or more objects in the environment; or description of respective states of one or more components of the robot.Clause 22. The method of any one of clauses 19-21, when dependent on clause 2, further comprising: obtaining the description of the current state of the environment by calling one or more of the APIs.Clause 23. The method of any one of clauses 19-22, wherein including the description of the current state of the environment in the instruction-based prompt causes the code generation neural network to include, in the output sequence, commented code describing one or more updated states of the environment.Clause 24. The method of any preceding clause, wherein the code generation neural network has been trained on a language modeling objective on a corpus of training computer code sequences from training computer programs.Clause 25. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method of any one of clauses 1-25.Clause 26. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any one of clauses 1-25.

Claims

CLAIMS1. A method for controlling a robot, the method comprising: obtaining a natural language instruction for interacting with the robot; generating an input sequence that comprises:(i) the natural language instruction; and(ii) an instruction-based prompt that comprises a natural language description of the robot and of the environment; processing the input sequence using a code generation neural network to generate as output a computer code sequence in a programming language that defines a computer program that represents a policy for controlling the robot; generating, from the computer code sequence, the computer program; and executing the computer program to control the robot.

2. The method of claim 1, wherein the computer code sequence comprises a call to one or more application programming interfaces (APIs), and wherein one or more of the APIs parameterize low-level control primitives for controlling the robot.

3. The method of claim 2, wherein the one or more APIS that parameterize low-level control primitives for controlling the robot comprise a first API that parameterizes a first low- level control primitive that maps a trajectory of waypoint locations for the robot to a sequence of control inputs for the robot that cause the robot to follow the trajectory of waypoint locations.

4. The method of claim 2 or claim 3, wherein one or more of the APIs can be called to obtain outputs from one or more sensors of the robot.

5. The method of any preceding claim, wherein the input sequence comprises (iii) a ir-shot prompt that comprises k examples, wherein k is greater than or equal to one, and wherein each example comprises an example natural language instruction and an example computer program that, when executed to control the robot, causes the robot carry out the example natural language instruction.

6. The method of claim 5 when dependent on claim 2, wherein the example computer programs in the k examples include calls to the one or more APIs.

7. The method of any one of claims 2-6, when dependent on claim 2 wherein the instruction-based prompt comprises text documenting the one or more APIs.

8. The method of any preceding claim, wherein executing the computer program to control the robot comprises: calling an execution function for the programming language with an input string derived from the computer code sequence.

9. The method of claim 8, when dependent on claim 2, wherein calling the execution function further comprises: calling the execution function with:(i) a global dictionary that specifies at least the one or more APIs, and(ii) a local dictionary that can be populated with at least variables defined during the execution.

10. The method of claim 9, the method further comprising: after executing the computer program, providing one or more values from the local dictionary in response to the natural language instruction.

11. The method of any preceding claim, wherein the code generation neural network is an auto-regressive neural network that auto-regressively generates each computed code token in the computer code sequence conditioned on the input sequence.

12. The method of claim 11, wherein the code generation neural network is a Transformer neural network.

13. The method of any preceding claim, wherein the natural language instruction specifies an instance of a task to be performed by the robot and wherein the instruction-based prompt comprises a natural description of one or more of: an objective of the task, constraints on performing the task, or a context for the task.

14. The method of any preceding claim, wherein the natural language description of the robot and of the environment comprises: natural language description of a state space of the robot.

15. The method of any preceding claim, wherein the natural language description of the robot and of the environment comprises: natural language description of an action space of the robot.

16. The method of any preceding claim, wherein the natural language description of the robot and of the environment comprises: natural language description of an embodiment of the robot.

17. The method of any preceding claim, wherein the natural language description of the robot and of the environment comprises: natural language description of one or more robot constraints.

18. The method of any preceding claim, wherein the natural language description of the robot and of the environment comprises: natural language description of one or more environment constraints.

19. The method of any preceding claim, wherein the instruction-based prompt further comprises: description of a current state of the environment.

20. The method of claim 19, wherein the description of the current state of the environment is formatted as computer code comments in the instruction-based prompt.

21. The method of claim 19 or 20, wherein the description of the current state of the environment comprises one or more of: description of positions of one or more objects in the environment; or description of respective states of one or more components of the robot.

22. The method of any one of claims 19-21, when dependent on claim 2, further comprising: obtaining the description of the current state of the environment by calling one or more of the APIs.

23. The method of any one of claims 19-22, wherein including the description of the current state of the environment in the instruction-based prompt causes the code generation neural network to include, in the output sequence, commented code describing one or more updated states of the environment.

24. The method of any preceding claim, wherein the code generation neural network has been trained on a language modeling objective on a corpus of training computer code sequences from training computer programs.

25. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1 -24.

26. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any one of claims 1-24.

Images