Controlling Agents Using Reporter Neural Networks

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for controlling an agent using a reporter neural network.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 410,156, filed September 26, 2022. The disclosure of that prior application is considered part of, and is incorporated by reference in, the disclosure of this application.

[0002] This specification relates to processing data using machine learning models. [Background technology]

[0003] A machine learning model receives inputs and generates outputs, e.g., predicted outputs, based on the received inputs. Some machine learning models are parametric models, which generate outputs based on the received inputs and the values ​​of the parameters of the model.

[0004] Some machine learning models are deep models that utilize multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to the received input to generate an output. Summary of the Invention [Means for solving the problem]

[0005] This specification generally describes a system implemented as a computer program on one or more computers at one or more locations that controls agents that interact with an environment and perform tasks in that environment.

[0006] Specifically, the system controls the agents using a behavior selection neural network system that includes a policy neural network, a planner neural network, and a reporter neural network.

[0007] This specification also describes training an action selection neural network system, and specifically, a reporter neural network.

[0008] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0009] This specification describes controlling agents using a planner neural network that generates natural language instructions and a policy neural network that receives natural language instructions and current observations and generates outputs that define actions to be performed by the agents.

[0010] For example, the planner neural network can be a pre-trained language model neural network, such as a large-scale language model neural network (LSLM).

[0011] Accomplishing complex tasks in a variety of different environments, such as embodied environments, often requires logical reasoning. For example, logical reasoning about the relationships between objects in the environment, about the properties of the environment or objects in the environment, or about the effects of an agent's actions on the environment continues to be a challenge for machine learning. Logical reasoning becomes even more challenging for embodied agents, where the agent must perceive and control objects or itself in its environment in addition to reasoning about how to accomplish a complex task.

[0012] On the other hand, recent language models have shown more promise for reasoning. However, these models are not embodied or grounded. They have no way to directly take action in an embodied environment or know what is happening in the environment. Therefore, using a language model neural network directly as a planner neural network has proven difficult.

[0013] To address these issues, the described techniques utilize a reporter neural network that can generate natural language reports characterizing an agent's progress in performing a task and provide these natural language reports as part of the input to a planner neural network. By giving the planner neural network access to this information, the system can effectively provide the planner neural network with the context it needs to generate effective instructions to the policy neural network, thereby improving how well the policy neural network can control the agent to perform the task. In other words, by incorporating a reporter neural network into a scheme for controlling agents, the system improves how well agents perform a variety of tasks, especially when the tasks require reasoning about aspects of the environment.

[0014] More specifically, because the planner neural network does not have the ability to interrogate the environment to observe the effects of actions taken by the agents, the natural language reports generated by the reporter neural network provide important information about the environment that the planner neural network can use to adjust the instructions it issues to the policy neural network (and thus to the agents), and that would not otherwise be available to the planner neural network.

[0015] Additionally, when the planner neural network is a pre-trained language model and the received observations are images or other sensor measurements of the environment, the planner and policy network do not operate over the same input space. That is, the policy network receives observations, e.g., images or other sensor measurements, and generates policy outputs that define actions, while the planner neural network receives natural language inputs (task descriptions and natural language reports) and generates natural language outputs (generated natural language instructions). While the policy network is conditioned on language and can interpret the planner's instructions, the planner cannot analyze the consequences of the actor's actions (to generate appropriate next actions) because it operates only on text inputs and not on observations that characterize, for example, the state of the environment. To compensate for this deficiency, the reporter neural network translates from the agent's action and observation space to the planner's input space via natural language reports generated using the reporter neural network.

[0016] Additionally, this specification describes techniques for training the reporter neural network, e.g., through reinforcement learning or imitation learning, or both. Specifically, the system can train the reporter neural network to learn what information is most useful to the planner, eventually converging to report only the truly important information to the planner at any given time step. This eliminates the need to specify in advance what information is important to the planner for any given task, and allows the system to adapt and generalize to new tasks that require different types of information to be provided to the planner in order for the planner to effectively control the agents.

[0017] In general, by incorporating a reporter neural network and a planner neural network in the loop in addition to a policy neural network, the resulting system can effectively perform any of a variety of tasks that require a combination of skills, such as two or more in the following examples. For example, the system can effectively perform tasks that require logical reasoning, i.e., the ability to receive complex instructions and perform various types of logical operations on the instructions to determine the correct course of action. As another example, the system can effectively perform tasks that require generalization, i.e., the ability to generalize beyond the agent's previous experience. As another example, the system can effectively perform tasks that require exploration, i.e., the ability to explore the world around the agent to uncover new information that can influence the reasoning about which actions to take. As another example, the system can effectively perform tasks that require perception, i.e., the ability to process the world using the agent's raw observations (usually visual) and use what it sees to make decisions.

[0018] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 illustrates an exemplary behavior selection system. [Figure 2] 1 is a flow diagram of an exemplary process for controlling an agent at a given time step. [Figure 3] 1 is a flow diagram of an exemplary process for training a reporter neural network through reinforcement learning. [Figure 4] FIG. 10 is a diagram illustrating an example of the operation of the behavior selection neural network system. [Figure 5]FIG. 1 illustrates an example of the improvement in agent control achieved by incorporating a planner neural network and a reporter that acts as a bridge between the policy neural network and the planner neural network. DETAILED DESCRIPTION OF THE INVENTION

[0020] Like reference numbers and designations in the various drawings indicate like elements.

[0021] 1 illustrates an exemplary behavior selection system 100. The behavior selection system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations where the systems, components, and techniques described below are implemented.

[0022] The behavior selection system 100 uses a behavior selection neural network system 102 to control an agent 104 that interacts with an environment 106 to accomplish a task by selecting a behavior 108 to be performed by the agent 104 at each of multiple time steps during the execution of an episode of the task.

[0023] Examples of agents, environments, and tasks are described below.

[0024] An "episode" of a task is a series of interactions during which an agent attempts to perform an instance of the task starting from some starting state of the environment. In other words, each task episode begins with the environment in an initial state, e.g., a fixed or randomly selected initial state, and ends when the agent successfully completes the task or when some termination criterion is met, e.g., the environment enters a state designated as an end state, or the agent performs a threshold number of actions without successfully completing the task.

[0025] Prior to the execution of a task episode, the system 100 receives a task description 112 that defines the tasks that need to be performed during the task episode. Specifically, the task description 112 is a natural language description of the task to be performed by the agent 104 in the environment 106. For example, the system 100 may receive the task description 112 as text or audio input from a user, which the system then converts into text.

[0026] Therefore, an additional constraint on a task episode may be that the episode continues over a series of time steps until one of the above criteria is met or until a new task description is received that specifies a new task to be performed (or modifies an existing task).

[0027] At each time step during any given task episode, the system 100 receives observations 110 that characterize the current state of the environment 106 at the time step, and in response selects actions 108 to be performed by the agent 104 at the time step.

[0028] Observations 110 may include any suitable information characterizing the state of the environment. As an example, observations 110 may include sensor measurements from one or more sensors configured to sense the environment. For example, observations 110 may include one or more images taken by one or more cameras, measurements from one or more proprioceptive sensors, etc.

[0029] After the agent performs a behavior 108, the environment 106 transitions to a new state.

[0030] In some cases, the system 100 receives a reward 150 from the environment in response to the agent performing a behavior.

[0031] In general, the reward is a scalar numerical value that characterizes the agent's progress in completing the task.

[0032] As one particular example, the reward may be a sparse binary reward that is 0 unless the action taken results in successful completion of the task, and 1 if the task is successfully completed.

[0033] As another particular example, the reward may be a dense reward that measures the agent's progress toward completing the task at the time of each observation received during an episode of attempting to perform the task, i.e., a non-zero reward may be received, and is received frequently, before successfully completing the task.

[0034] While performing any given task episode, the system 100 selects actions in an attempt to maximize the returns received over the course of the task episode.

[0035] That is, at each time step during the episode, the system 100 selects an action that attempts to maximize the return received for the remainder of the task episode starting at the time step.

[0036] In general, at any given time step, the return received is a composite of the rewards received at time steps that follow the given time step in the episode.

[0037] For example, at a given time step t, the return is Σ i gamma i-t-1 r i where i spans either all time steps after t in the episode or some fixed number of time steps after t in the episode, γ is a discount factor, and r i is the reward at time step i.

[0038] As described above, the system 100 uses the behavior selection neural network system 102 to control the agent 104 .

[0039] The behavior selection system 102 includes a policy neural network 120 , a planner neural network 130 , and a reporter neural network 140 .

[0040] The policy neural network 120 is a neural network configured to process observations and natural language instructions 132 as inputs to generate a policy output 122 that defines an action to be taken by the agent. The system 100 then uses the policy output 122 to select an action 108 to be taken by the agent 104.

[0041] Policy neural network 120 may have any suitable architecture that enables policy neural network 120 to map observations and natural language instructions to policy outputs.

[0042] For example, the policy neural network 120 may include one encoder for observations, another encoder for natural language instructions, and a policy sub-network configured to process the outputs from the two encoders to generate a policy output. In some cases, the policy sub-network may include a memory neural network, such as a recurrent neural network or a Transformer neural network, so that the policy output 122 at a given time step incorporates information from the previous time step.

[0043] As a particular example, when neural network 120 receives a visual observation, e.g., an image or video, the neural network may have a convolutional visual encoder to encode the visual observation and a recurrent neural network, e.g., an LSTM-based linguistic encoder, to encode the action instructions. The neural network may also have an LSTM-based memory module to help consider previous actions and observations for the policy output.

[0044] In one example, policy output 122 may include a respective Q-value for each action in the fixed set. System 100 may process the Q-values ​​(e.g., using a soft-max function) to generate a respective probability value for each action that may be used to select an action, or may select the action with the highest Q-value.

[0045] The Q-value for an action is an estimate of the "payoff" that results from the agent performing the action in response to the current observations and then being controlled using the actions generated by the action selection system.

[0046] In another example, the policy output 122 may include a respective numerical probability value for each action in the fixed set. The system 100 may select an action by, for example, sampling the actions according to the probability values ​​and selecting the action with the highest probability value.

[0047] As another example, when the action space is continuous, the policy output 122 may include parameters of a probability distribution over the continuous action space. The system 100 can then select an action by sampling actions from the probability distribution or by selecting the average action.

[0048] Planner neural network 130 is a neural network configured to process planner inputs and generate natural language instructions 132 for policy neural network 120 as output.

[0049] Generally, the planner input includes a task description 112 and a natural language report 142, which, as explained in more detail below, describes the current state of the task episode. That is, the natural language report 142 at a given time step during the task episode characterizes the agent's progress toward completing the task as of the given time step.

[0050] The natural language instructions can be thought of as a "plan" output that provides high-level information to the policy neural network 120 that guides it to select appropriate "low-level" actions useful in performing a task, i.e., to control the agent.

[0051] For example, the planner neural network 130 may be a language model neural network that has been trained for language modeling purposes. That is, the language model neural network may have been pre-trained for language modeling purposes on a large corpus of training data. In some implementations, the planner neural network 130 is not trained to control agents, i.e., it is not trained together with either the policy neural network 120 or the reporter neural network 140.

[0052] A language model neural network is a neural network configured to process an input and generate an output that includes a probability distribution over a set of text tokens in a vocabulary of text tokens, where the probability for each token represents the likelihood that the text token immediately follows the input.

[0053] For example, the language model neural network may be an autoregressive language model neural network.

[0054] A language model neural network is called an autoregressive neural network because it autoregressively generates an output sequence of tokens by generating each particular token in the output sequence conditional on the current input sequence including any token that precedes the particular text token in the output sequence, i.e., a token already generated for any previous position in the output sequence that precedes the particular position of the particular token, and contextual input (a "context sequence") that provides the context for the output sequence.

[0055] For example, the current input sequence when generating a token at any given position in the output sequence may include a context sequence and a token at any preceding position that precedes the given position in the output sequence. As a specific example, the current input sequence may include a context sequence followed by a token at any preceding position that precedes the given position in the output sequence. Optionally, the context sequence and the current output sequence may be separated by one or more predetermined tokens within the current input sequence.

[0056] More specifically, to generate a particular token at a particular position in a candidate output sequence, the neural network can process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each text token in a vocabulary of text tokens. The neural network can then use the score distribution to select a text token from the vocabulary as the particular token. For example, the neural network can greedily select the highest-scoring token or can sample a token from the distribution, e.g., using kernel sampling or another sampling technique.

[0057] As a particular example, the language model neural network may be an autoregressive Transformer-based neural network that includes (i) multiple attention blocks, each applying a self-attention operation, and (ii) an output subnetwork that processes the output of the last attention block to generate a score distribution.

[0058] This is the ultimate Transformer sledgehammer [ PubMed ] [ Cross Ref ] J. Hoffmann S. Bo rgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, DdLCasas, LAHendricks, J. Welbl, and A. Clark, Training Compute-Optimal Large Language Models、arXiv Preprint arXiv:2203.15556, 2022, JWRae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HFSong, and J. Aslanides S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van then Driessche, LAHendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higg ins, A. Creswell, N. McAleese, A. Wu, E. Elsen, SM Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan. M. Paganini, L. Sifre, L. Martens, XLLi, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mens ch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, BAHechtman, L. Weidinger, I. Gabriel, WSIsaac, E. Lockhart, and S. S. Casas.Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving: Scaling language models: Methods, analysis & insights from training gopher, CoRR, abs / 2112.11446, 2021, Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Gopher Liu, arXiv preprint arXiv:1910.10683, 2019, Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le, Towards a Human-Like Open-Domain Chatbot, CoRR, abs / 2001.09977, 2020, and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, etc. Language Models Are Few-Shot Learners arXiv preprint arXiv:2005.14165 |

[0059] In general, however, a Transformer-based neural network includes a series of attention blocks, and during processing of a given input sequence, each attention block in the sequence receives a respective input hidden state for each input token in the given input sequence. The attention block then updates at least the hidden state for the last token in the given input sequence, at least in part by applying self-attention, to generate a respective output hidden state for the last token. The input hidden state for the first attention block is an embedding of the input token in the input sequence, and the input hidden state for each subsequent attention block is the output hidden state generated by the preceding attention block.

[0060] In this example, the output sub-network processes the output hidden state generated by the last attention block in the sequence for the last input token in the input sequence to generate a score distribution.

[0061] As described, before using the language model neural network as planner neural network 130, system 100 or another training system pre-trains the language model neural network for a language modeling task, e.g., a task that requires predicting the next token after the current sequence in the training data given a current sequence of text tokens. As a particular example, the language model neural network may be pre-trained for maximum likelihood on a large data set containing text, e.g., text publicly available from the Internet or another text corpus.

[0062] For example, reporter neural network 130 may be a pre-trained large-scale language model (LSLM). By way of example, the LSLTM may be one of two variants of the Chinchilla model described in Hoffmann et al., linked above, namely, a model with 70 billion parameters (referred to as the 70B model) and a model with 7 billion parameters (referred to as the 7B model).

[0063] In this example, after the first time step in the task episode, the planner input may also include reporter input from one or more previous time steps and natural language commands generated from one or more previous time steps. Including these previous outputs can provide the planner neural network 130 with context for generating appropriate natural language commands for the current time step.

[0064] In this example, the planner inputs may also include one or more prompt inputs that provide the planner with access to “ground truth” agent control examples in response to the example task description. That is, the planner inputs may include k-shot prompts that include k prompt inputs, where k is an integer greater than or equal to 1. For example, each planner input may include an example task description, one or more example reporter inputs, and, for each reporter input, an example natural language report that is generated in response to the reporter input. The inclusion of k-shot prompts may aid the planner neural network 130 in performing “in-context learning” to adapt to generating appropriate natural language instructions.

[0065] The reporter neural network 140 is a neural network configured to process reporter inputs including observations and generate reporter outputs that define natural language reports 142, i.e., reports 142 that characterize the agent's progress in completing the task as of that time step.

[0066] In some implementations, the reporter output is a natural language report 142. That is, the reporter neural network 140 is a generative neural network that processes observations to generate a natural language output.

[0067] For example, the reporter neural network 140 may have any suitable visual language model (VLM) neural network architecture. A visual language model (VLM) is a neural network that receives images and optionally text as input and generates text as output. For example, a VLM may include an encoder neural network, such as a Vision Transformer or a convolutional neural network, that processes images to generate an encoded representation of the image, and a decoder neural network, such as a Transformer-based decoder neural network, that generates text conditional on the encoded representation of the image and optionally input text embeddings. One example of a VLM is the Flamingo model described in *Flamingo: a Visual Language Model for Few-Shot Learning*, available at arXiv:2204.14198. Another example of a VLM is described in *Multimodal Few Shot Learning with Frozen Language Models*, available at arXiv:2106.13884.

[0068] In some other implementations, the reporter output may be a classification output across multiple categories, e.g., multiple categories each representing a different level of progress toward completing a task. As an example, when the task is to proceed to a specified location, each category may represent a different range of distance between the agent and the specified location. As another example, when the task is to find an object with specified properties, each category may represent a different property of the object in the environment closest to the agent. In these implementations, the system 102 may generate the report 142 by using the classification output to select a category from the multiple categories, for example, by sampling from the classification output, or by generating the natural language report by selecting the category with the highest score according to the classification output and inserting a natural language description of the selected category into a predetermined location in the natural language report.

[0069] In these implementations, the reporter neural network 140 may have any suitable architecture that enables the neural network to match observations with classification outputs, for example, a convolutional neural network, a fully connected neural network, or a self-attention-based neural network.

[0070] For example, when the observations include images, the reporter neural network 140 may include a first encoder sub-network, such as a vision transformer or a convolutional neural network, that encodes the images.

[0071] In some implementations, one or more of the encoder neural networks are pre-trained, for example, as part of a classification neural network or for a representation learning task. As a particular example, when the observations include images of an environment and the reporter neural network includes a first encoder neural network configured to process the images, the first encoder may be pre-trained for a visual representation learning task.

[0072] As another example, when the reporter includes natural language instructions (i.e., in addition to the observations) from one or more prior time steps, the reporter neural network may include a second encoder neural network configured to process the natural language instructions and encode the instructions. In some of these cases, the second encoder may be pre-trained on a text representation learning task.

[0073] The use of a pre-trained encoder may enable the system 100 to leverage learned representations of large-scale data, which may lead to the generation of accurate reports by the reporter neural network.

[0074] In some implementations, policy neural network 120 and reporter neural network 140 each receive the same type of observations, i.e., each receive observations 110 .

[0075] In some other implementations, policy neural network 120 and reporter neural network 140 receive different types of observations that contain different information about the environment. For example, when observations 110 include data generated by multiple sensors, policy neural network 120 can receive second observations that include data from a second subset of sensors, while reporter neural network 140 receives first observations that include data from a first subset of sensors.

[0076] In some implementations, the reporter neural network 140 may include a memory sub-network, such as a recurrent neural network or a Transformer neural network, so that the reporter neural network 140 can incorporate context from previous time steps.

[0077] In some implementations, reporter input includes additional information in addition to observations, such as actions taken in one or more previous time steps.

[0078] In general, at any given time step during a task episode, the system 100 uses the reporter neural network 140 and the planner neural network 130 to generate inputs to the policy neural network 120, and then uses the policy output 122 generated by the policy neural network 120 to select the action 108 to be performed by the agent 104 at that time step.

[0079] The system 100 then causes the agent 104 to perform the selected action 108 .

[0080] This is explained in more detail below with respect to FIG.

[0081] 2 is a flow diagram of an exemplary process 200 for controlling an agent at a given time step during a task episode. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, a behavior selection system suitably programmed in accordance with this specification, such as behavior selection system 100 of FIG. 1, can perform process 200.

[0082] The system obtains a first observation that characterizes the state of the environment at that time step (step 202).

[0083] The system processes the reporter input, including the first observation, using a reporter neural network to generate a reporter output (step 204).

[0084] In general, the reporter output defines a natural language report that characterizes the agent's progress in completing the task as of that time step.

[0085] In some cases, reporter input may also include additional information, such as a task description, natural language instructions provided to the agent in a previous time step, actions performed in one or more previous time steps, etc.

[0086] The system processes planner inputs, including task descriptions and natural language reports, using a planner neural network to generate natural language instructions for the agents (step 206).

[0087] The system obtains a second observation characterizing the state of the environment at that time step (step 208).

[0088] In some cases, the first observation and the second observation are the same, for example, both may include information generated from sensor measurements produced by the same set of one or more sensors sensing the environment at that time step.

[0089] In some other cases, the first observation and the second observation may include different information, for example, the first observation and the second observation may include information generated from sensor measurements generated by different combinations of sensors sensing the environment at that time step.

[0090] The system processes the natural language instruction and the second observation using a policy neural network to generate a policy output (step 210). As explained above, the policy output defines the action to be performed by the agent.

[0091] The system uses the policy output to control the agent (step 212), ie, by using the policy output to select an action and causing the agent to perform the selected action.

[0092] As explained above, as a result of the agent performing a selected action, the environment typically changes state and the system receives the next observation for the next time step that characterizes the new state of the environment.

[0093] The system can then use the next observation and the reporter neural network to generate a new natural language report for the next time step.

[0094] Optionally, the system can then receive a reward indicative of the agent's progress in completing the task.

[0095] Thus, because the planner neural network does not have the ability to interrogate the environment to observe the effects of actions taken by the agents, the natural language reports generated by the reporter neural network provide important information about the environment that the planner neural network can use to adjust the instructions it issues to the policy neural network (and therefore to the agents).

[0096] Additionally, when the planner neural network is a pre-trained language model and the received observations are images of the environment or other sensor measurements of the environment, the planner and policy network do not operate over the same observation space: the policy network operates over pixel observations (or other sensor measurements) to generate actions, while the planner operates over verbal observations (prompts) to generate verbal actions (commands to be generated).

[0097] The policy network is conditional on language and can interpret the planner's commands, but the planner cannot analyze the consequences of the actor's actions (to generate appropriate next actions). Thus, the reporter transforms from the space of the actor's actions and observations to the space of the planner's actions and observations.

[0098] In some implementations, the planner and reporter execute for less than every time step in a given task episode. For example, the planner and reporter may execute every n time steps, where n is a fixed integer greater than 1. In these implementations, the policy neural network may continue to receive the same natural language instructions at different time steps until new natural language instructions are generated by the planner.

[0099] When the system receives a reward in response to performing an action, the system can use the received reward to train a reporter neural network through reinforcement learning, i.e., to maximize the expected reward received in response to an action performed by the agent.

[0100] This is illustrated in FIG.

[0101] 3 is a flow diagram of an exemplary process 300 for training a reporter neural network. For convenience, process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a behavior selection system appropriately programmed in accordance with the present application, such as behavior selection system 100 of FIG. 1, can perform process 300.

[0102] The system controls the agents to perform actions using a reporter neural network, a policy neural network, and a planner neural network (step 302), for example, as described above with respect to FIG.

[0103] The system receives a reward as a result of the agent performing the action (step 304).

[0104] The system trains the reporter neural network using the rewards through reinforcement learning (step 306).

[0105] For example, the system can store transitions, including observations, task descriptions, natural language reports, and rewards, in a replay memory. The system can then repeatedly sample transitions or sequences of transitions from the replay memory and use the sampled transitions to train a reporter neural network through reinforcement learning.

[0106] The system may use any suitable reinforcement learning training technique to train the reporter neural network. Examples of reinforcement learning techniques that may be used include V-Trace, PPO, MPO, etc.

[0107] As a specific example, the system can train the reporter neural network while holding the planner neural network and the policy neural network unchanged. In this example, although rewards were received as a result of the agent being controlled using the policy output, the system did not use the rewards to train the policy neural network; instead, it used the rewards only to train the reporter neural network, which generated natural language reports that only "indirectly" selected actions (and therefore rewards) by changing how the planner neural network was conditioned.

[0108] Specifically, the planner neural network and the policy neural network may be pre-trained for their respective purposes, and the system can keep the parameters of the planner and the policy neural network fixed at the values ​​generated as a result of this pre-training.

[0109] For example, the policy neural network may have been trained through reinforcement learning or imitation learning on one or more different tasks other than those specified by the task description. For example, the policy neural network may have been trained using a task description generated by a user or by a different planner neural network. Additionally, as described above, the planner neural network may have been pre-trained on a language modeling task and kept fixed during reporter training (and not used during policy neural network training).

[0110] Thus, as reinforcement learning training of the reporter neural network continues, the system can use rewards, such as the final reward of an episode for a sparse reward task, to reinforce learning which information is most useful to the planner neural network, until the reporter training eventually converges to reporting only the true and important information to the planner at any given time step. This eliminates the need to specify in advance what information is important to the planner for any given task, and allows the system to adapt and generalize to new tasks that require different types of information to be provided to the planner in order for it to effectively control the agent.

[0111] In some cases, instead of or in addition to training the reporter neural network through reinforcement learning, the system can obtain a set of demonstration data including a series of observation-action pairs that characterize the interaction of the expert agent with the environment to perform corresponding tasks, and for each corresponding task, a natural language description of the task. The system can then train the reporter neural network on this demonstration data through imitation learning, e.g., through behavior cloning or adversarial imitation learning.

[0112] Additionally, as described above, in some implementations, the reporter neural network may have one or more encoders that have been pre-trained for other tasks. In these implementations, the system can either (i) keep these encoders fixed during the above-described training of the reporter neural network and train only the rest of the reporter neural network, or (ii) fine-tune the pre-trained encoders along with the rest of the reporter neural network.

[0113] FIG. 4 illustrates an example 400 of the operation of the system when controlling agents to perform task episodes.

[0114] In the simple example in Figure 4, the task requires information gathering. The goal of the episode is to lift the correct object based on the secret properties of another object. The task description passed to the planner neural network is: "If {Decisor object} is good, lift {Object 1}, otherwise lift {Object 2}."

[0115] Successful completion of an episode of this task requires five high-level steps: a) the planner commands the actor to examine {decision-maker object}, b) the actor examines the object, c) the reporter relays the revealed information, d) the planner reasons which object should be chosen based on the reporter, {object 1}, or {object 2} and commands the actor to lift the correct object, and e) the actor lifts the correct object.

[0116] In the example of Figure 4, the "plain blue circle" is the decision maker object, the checkered green triangle is object 1, while the striped yellow cross is object 2. Thus, in the example of Figure 4, the task description 402 for the episode is "If the plain blue circle is good, lift the checkered green triangle; otherwise, lift the striped yellow cross."

[0117] To control the agent, the actor 410 implements an instance of the policy neural network and acts in the environment in response to observations received using the policy neural network. At a given time step, the system uses the reporter neural network 140 to generate a natural language report 420: "I looked at the solid blue circle. Its secret property is 'good'."

[0118] The system provides this report 420 along with the task description 402 to the planner neural network 130, which generates natural language instructions 430 to have the agent "investigate the blue circle."

[0119] The system provides natural language instructions 430 to the actor 410, who uses the instructions 430 and observations to generate inputs to the policy neural network 120.

[0120] Thus, reports 430 provide information to planner neural network 130 about the agent's progress in performing the task, for example, which objects the agent has examined so far and what their properties are. Planner neural network 130 can then use this information to update the instructions provided to the agent to have it examine important objects so as to successfully complete the task.

[0121] FIG. 5 shows an example 500 of the improvements in agent control achieved by incorporating a planner neural network and a reporter that acts as a bridge between the policy neural network and the planner neural network.

[0122] Specifically, part A of Figure 5 shows the performance on the information gathering task using a) a 7B parameter planner neural network, b) a 70B parameter planner neural network, and c) a "trained RL baseline" that does not use a reporter neural network or a planner neural network, but instead provides "reports" such as those that would be generated by the reporter neural network directly to the policy neural network.

[0123] Specifically, the example in Figure 5 shows performance on a "conditional" information-gathering task and a "search" information-gathering task, such as those described above. The "search" task extends the "conditional" task by requiring an additional step of information gathering. Instead of examining a single object, the agent must examine multiple objects, note their secret properties, and pick up the correct object for a reward. For example, a task description might specify, "The objects are {}, {}, {}, {}, and {}. Pick up the object with the secret property 'good'." A successful episode consists of the planner asking the actor to examine each object in turn until the actor finds one with the property "good," at which point the planner asks the actor to pick up the object.

[0124] Part A of Figure 5 shows that both the 7B parameter planner and the 70B parameter planner outperform the “trained RL” baseline on both tasks, with the 70B parameter planner (and the commensurate improvement in reasoning ability encoded within the parameters of the larger planner neural network) significantly outperforming the baseline.

[0125] As can be seen from the baseline, both of these are difficult tasks for the reinforcement learning baseline, even when the policy neural network is given the "report" as an additional input.

[0126] In some cases, reporters may provide "noisy" information to the planner: due to the variable nature of the task, and because accurately describing an agent's progress is a difficult task, natural language reports may sometimes be erroneous or not important to the plan.

[0127] Part B of Figure 5 shows the robustness of variants 7B, 70B, and the baseline to unimportant reports, where, for example, reports are forced to describe unimportant actions 20% of the time. As shown in part B, both variants 7B and 70B outperform the baseline in the presence of unimportant reports. Moreover, when k-short prompts to the planner describe how to handle unimportant reports, performance returns to almost the original level (where unimportant reports are reduced).

[0128] Part C shows the increasing training reward achieved as the reporter neural network is trained on the vision-conditioned task while holding the planner and policy neural networks fixed. As can be seen from Part C, training the reporter neural network significantly improves the performance of the policy neural network in controlling the agent.

[0129] An example of a visually contingent task is one in which a target object changes depending on the color of the agent. This agent color information is not available to the language model, i.e., the planner. Therefore, the reporter must learn to decode it from the visual observation and report it as a response. The language model can then issue the correct next command, which leads to a reward.

[0130] A more complex example of a vision-conditioned task that combines the challenges of a multi-step task with learning to report is one in which the agent must perform information-gathering behaviors to obtain important information. For example, which object the agent must choose may depend on whether a designated decision-maker object is close to a wall or other location in the environment. To determine this, the agent must move toward the object and examine its surroundings.

[0131] As explained above, including a reporter neural network can improve the operation of the system in various ways, even when the reporter provides noisy information to the planner.

[0132] Some examples of this are shown in Table 1 below, where the reporter output is denoted Agent and the planner output is denoted LM. The first box highlights an error made by the reporter, and the second box shows how the planner iterates the policy neural network and the planner to report, resulting in the correction of the error.

[0133] [Table 1]

[0134] In some cases, the planner neural network can be made to repeat commands, for example by k-shot prompts, in an attempt to get the reporter to provide important information rather than unimportant or noisy information. This is shown in Table 3 below.

[0135] [Table 2]

[0136] In some implementations, the planner can repeat the command, for example to check different objects, until a good object report is received, as shown in Table 3 below.

[0137] [Table 3]

[0138] In some implementations, the environment is a real-world environment, the agent is a mechanical agent that interacts with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle that operates or navigates within the environment, and the behavior is an action taken by the mechanical agent in the real-world environment to perform a task. For example, the agent may be a robot that interacts with the environment to accomplish a particular task, e.g., to find an object of interest in the environment, or to move an object of interest to a specified location in the environment, or to navigate to a specified destination in the environment.

[0139] In these implementations, the observations may include, for example, one or more of images, object position data, and sensor data to capture observations as the agent interacts with the environment, e.g., from image, distance, or position sensors, or from actuators. For example, in the case of a robot, the observations may include data characterizing the current state of the robot, e.g., one or more of joint positions, joint velocities, joint forces, torques, or accelerations, e.g., gravity-compensated torque feedback, and the global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle, the observations may similarly include one or more of positions, linear or angular velocities, forces, torques, or accelerations, and the global or relative pose of one or more parts of the agent. The observations may be defined in one, two, or three dimensions and may be absolute and / or relative observations. Observations may also include, for example, sensed electronic signals such as motor current or temperature signals, and / or image or video data, for example from a camera or LIDAR sensor, e.g., data from a sensor on the agent or from a sensor located separately from the agent in the environment.

[0140] In these implementations, a behavior may be a control signal for controlling a robot or other mechanical agent, e.g., torque or high-level control command for a robot's joints, or a control signal for controlling an autonomous or semi-autonomous land, air, or sea vehicle, e.g., torque or high-level control command for a control surface or other control element, e.g., a steering control element of the vehicle. The control signal may include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or part of another mechanical agent. The control signal may also or instead include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices in the environment, the control of which has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, the control signal may define actions for controlling navigation, e.g., steering, and actions for controlling movement, e.g., braking and / or accelerating the vehicle.

[0141] In some implementations, the environment is a simulation of the real-world environment described above, and the agent is implemented as one or more computers that interact with the simulated environment. For example, the simulated environment may be a simulation of a robot or vehicle, and the reinforcement learning system may be trained on the simulation and, once trained, used in the real world.

[0142] In some implementations, the environment is a real-world manufacturing environment for producing a product, such as a chemical, biological, or mechanical product, or a food product. As used herein, "manufacturing" a product also includes refining raw materials to create a product or processing raw materials, e.g., to remove contaminants and produce a clean or recycled product. A manufacturing plant may include multiple manufacturing units, such as vessels for chemical or biological substances or machines, e.g., robots, for processing solids or other materials. The manufacturing units are configured such that intermediate versions or components of the product can be moved between manufacturing units during the production of the product, e.g., via piping or mechanical transport. As used herein, manufacturing a product also includes the production of food by a kitchen robot.

[0143] The agent may comprise an electronic agent configured to control a manufacturing unit or a machine, such as a robot, that operates to manufacture the product. That is, the agent may comprise a control system configured to control the manufacturing of a chemical, biological, or mechanical product. For example, the control system may be configured to control one or more of the manufacturing units or machines, or to control the movement of intermediate versions or components of the product between the manufacturing units or machines.

[0144] As one example, the tasks performed by the agents may comprise tasks for manufacturing a product or intermediate versions or components thereof. As another example, the tasks performed by the agents may comprise tasks for controlling, e.g., minimizing, resource usage, such as tasks for controlling power consumption, or water consumption, or consumption of any materials or consumables used in the manufacturing process.

[0145] The actions may comprise control actions for controlling the use of a machine or production unit to process solid or liquid materials to manufacture a product or an intermediate version or component thereof, or for controlling the movement of an intermediate version or component of a product within a manufacturing environment, for example, between production units or machines. In general, the actions may be any action that affects the observed state of the environment, for example, an action configured to adjust any of the sensed parameters described below. These may include actions for adjusting the physical or mechanical conditions of a production unit or for controlling the movement of mechanical parts of a machine or joints of a robot. The actions may include actions that impose operating conditions on a production unit or machine, or actions that effect changes in settings to adjust, control, or turn on or off the operation of a production unit or machine.

[0146] The reward or return may relate to a measure of the performance of the task. For example, in the case of a task to manufacture a product, the measure may comprise a measure of the quantity of product manufactured, the quality of the product, the speed at which the product is manufactured, or a measure of the physical cost of performing the manufacturing task, e.g., the amount of energy, material, or other resource used to perform the task. In the case of a task to control the use of a resource, the measure may comprise any measure of resource usage.

[0147] In general, the observations of the state of the environment may comprise any electronic signal representative of the function of electronic and / or mechanical equipment. For example, the representation of the state of the environment may be derived from observations made by sensors that detect the state of the manufacturing environment, e.g., sensors that detect the state or configuration of a manufacturing unit or machine, or sensors that detect the movement of materials between manufacturing units or machines. As some examples, such sensors may be configured to detect mechanical movement or force, electrical conditions such as pressure, temperature, current, voltage, frequency, impedance, etc.; quality, level, flow rate / movement speed or flow path / movement path of one or more materials; physical or chemical conditions, e.g., physical state, shape, or configuration, or chemical conditions such as pH; configurations of units or machines, such as the mechanical configuration of a unit or machine, or configurations of valves; may be image or video sensors for capturing images or video observations of a manufacturing unit or machine or movement; or any other suitable type of sensor. In the case of a machine such as a robot, observations from sensors may include observations of the position, linear or angular velocity, force, torque or acceleration, or pose of one or more parts of the machine, e.g., data characterizing the current state of the machine or robot, or of an item being held or processed by the machine or robot. Observations may also include sensed signals, e.g., motor current or temperature signals, or image or video data, e.g., from a camera or LIDAR sensor. Sensors such as these may be part of the agent or may be located separately from it in the environment.

[0148] In some implementations, the environment is a real-world environment of a service facility comprising multiple electronic devices, such as a server farm or data center, e.g., a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include auxiliary controls that control the operating environment of multiple devices, e.g., environmental control devices such as temperature controls, airflow control or air conditioning devices, e.g., heaters, coolers, humidifiers, or other hardware that alters the properties of the air in the real-world environment. The tasks may comprise tasks to control, e.g., minimize, resource usage, such as tasks to control power consumption or water consumption. The agents may comprise electronic agents configured to control the operation of the devices or to control the operation of auxiliary controls, e.g., environmental control devices.

[0149] In general, the actions may be any actions that affect the observed state of the environment, for example, actions configured to adjust any of the sensed parameters described below. These may include actions to control or impose operating conditions on the appliance or auxiliary control, for example, actions that effect a change in settings to adjust, control, or turn on or off the operation of an appliance or auxiliary control.

[0150] In general, observations of the state of the environment may comprise any electronic signal that represents the function of the facility or equipment within the facility. For example, the representation of the state of the environment may be derived from observations made by any sensors that sense the state of the facility's physical environment, or from observations made by any sensors that sense the state of one or more pieces of equipment or one or more auxiliary controls. These include sensors configured to sense electrical conditions such as current, voltage, power, or energy, facility temperature, flow rate, temperature or pressure within the facility or within the facility's cooling system, or physical facility configuration such as whether a vent is open.

[0151] The reward or return may relate to a measure of performance of the task. For example, in the case of a task to control, e.g., minimize, resource usage, such as a task to control power or water usage, the measure may comprise any measure of resource usage.

[0152] In some implementations, the environment is a real-world environment of a power generation facility, e.g., a renewable power generation facility such as a solar or wind power plant. The tasks may comprise control tasks for controlling power generated by the facility, e.g., for controlling the delivery of power to a power grid, e.g., for meeting demand or reducing the risk of mismatches between grid elements, or for maximizing power generated by the facility. The agents may comprise electronic agents configured to control the generation of power by the facility or the coupling of generated power to a grid. The actions may comprise actions for controlling the electrical or mechanical configuration of a generator, such as the electrical or mechanical configuration of one or more renewable power generation elements, e.g., controlling the electrical or mechanical configuration of a wind turbine or a solar panel or a plurality of solar panels or mirrors, or a rotating generator. The mechanical control actions may comprise, e.g., actions for controlling the conversion of an energy input to an electrical energy output, e.g., the efficiency of the conversion of the energy input to the electrical energy output or the degree of coupling. The electrical control actions may comprise, e.g., actions for controlling one or more of the voltage, current, frequency, or phase of the generated power.

[0153] The reward or return may relate to a measure of the performance of the task. For example, in the case of a task to control the delivery of power to a distribution grid, the measure may relate to a measure of the power delivered, or a measure of an electrical mismatch between the power generation facility and the grid, such as a mismatch in voltage, current, frequency, or phase, or a measure of power or energy losses at the power generation facility. In the case of a task to maximize the delivery of power to a distribution grid, the measure may relate to a measure of the power or energy delivered to the grid, or a measure of power or energy losses at the power generation facility.

[0154] In general, environmental state observations may comprise any electronic signals representative of the electrical or mechanical function of generators in a power generation facility. For example, the representation of the environmental state may be derived from observations made by any sensors that sense the physical or electrical state of equipment in the power generation facility that is generating electrical power, or the state of the physical environment of such equipment, or auxiliary equipment supporting the generators. Such sensors may include sensors configured to sense the electrical state of the equipment, such as current, voltage, power, or energy; the temperature or cooling of the physical environment, flow rate, or physical configuration of the equipment; and observations of the electrical state of the grid, for example, from local or remote sensors. The environmental state observations may also comprise one or more predictions regarding future conditions for generator operation, such as a prediction of future wind levels or solar irradiation or a prediction of future grid electrical conditions.

[0155] As another example, the environment may be a chemical synthesis or protein folding environment, where each state is a respective state of a protein chain or one or more intermediate or precursor chemicals, and the agent is a computer system for determining how to fold a protein chain or synthesize a chemical. In this example, the behaviors are possible folding behaviors for folding a protein chain or behaviors for assembling precursor chemicals / intermediates, and the results to be achieved may include, for example, folding a protein so that the protein is stable and achieves a specific biological function, or providing an effective synthetic pathway for a chemical. As another example, the agent may be a mechanical agent that indirectly performs or controls protein folding behaviors, for example, by controlling chemical synthesis steps automatically selected by the system without human interaction. The observations may comprise direct or indirect observations of the state of the protein or chemicals / intermediates / precursors and / or may be derived from simulations. Thus, the system may be used to automatically synthesize proteins with specific functions, such as having a binding site shape, e.g., a ligand that binds with sufficient affinity to be used as a drug for a biological effect. For example, it may be a receptor or enzyme agonist or antagonist, or it may be an antibody that binds to an antibody target such as a viral coat protein, or a protein expressed on cancer cells, e.g., acting as an agonist of a specific receptor or configured to prevent the binding of another ligand and thus the activation of an important biological pathway.

[0156] Similarly, the environment may be a drug discovery environment where each state is a respective state of a potentially pharmacologically active compound, and the agent is a computer system for determining elements of pharmacologically active compounds and / or synthetic pathways for pharmacologically active compounds. Drugs / synthesis may be designed based on rewards derived from drug targets, for example, in a simulation. As another example, the agent may be a mechanical agent that performs or controls the synthesis of a drug.

[0157] In some further applications, the environment is a real-world environment and the agent manages the distribution of tasks across computational resources, e.g., on mobile devices and / or in a data center. In these implementations, the actions may include assigning tasks to specific computational resources.

[0158] As a further example, the action may include presenting an advertisement, the observation may include an advertisement impression or click-through count or rate, and the reward may characterize a previous selection of an item or content made by one or more users.

[0159] In some cases, the observations may include text or voice commands given to the agent by a third party (e.g., the operator of the agent. For example, the agent may be an autonomous vehicle, and a user of the autonomous vehicle may provide text or voice commands to the agent (e.g., to proceed to a particular location).

[0160] As another example, the environment may be an electrical, mechanical, or electromechanical design environment, e.g., an environment in which the design of an electrical, mechanical, or electromechanical entity is simulated. The simulated environment may be a simulation of a real-world environment in which the entity is intended to operate. The task may be to design the entity. The observations may comprise observations characterizing the entity, i.e., observations of the mechanical shape or of the entity's electrical, mechanical, or electromechanical configuration, or observations of the entity's parameters or characteristics. The actions may comprise actions that modify the entity, e.g., modify one or more of the observations. The compensation or reward may comprise one or more measures of performance of the entity's design. For example, the compensation or return may relate to one or more physical characteristics of the entity, such as weight or strength, or one or more electrical characteristics of the entity, such as a measure of efficiency in performing a particular function for which the entity is designed. The design process may include outputting a design for manufacturing, e.g., in the form of computer-executable instructions for manufacturing the entity. The process may include making the entity according to the design. Thus, the design of the entity may then be optimized, e.g., by reinforcement learning, as an optimized design output, e.g., computer-executable instructions, for manufacturing the entity. The entity may then be manufactured using the optimized design.

[0161] As previously described, the environment may be a simulated environment. Generally, in the case of a simulated environment, the observations may include simulated versions of one or more of the previously described observations, or the observations and action types may include simulated versions of one or more of the previously described actions or action types. For example, the simulated environment may be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent may be a simulated vehicle that navigates through the motion simulation. In these implementations, the actions may be control inputs for controlling a simulated user or a simulated vehicle. Generally, the agent may be implemented as one or more computers that interact with the simulated environment.

[0162] The simulated environment may be a simulation of a particular real-world environment and agent. For example, the system may be used to select actions in a simulated environment during training or evaluation of the system, and after training, evaluation, or both, is complete, the system may be deployed to control a real-world agent in the particular real-world environment that was the subject of the simulation. This may avoid unnecessary wear, tear, and damage to the real-world environment or real-world agent, and may allow the control neural network to be trained and evaluated for situations that occur rarely or are difficult or unsafe to replicate in the real world. For example, the system may be partially trained using a simulation of a mechanical agent in a simulation of a particular real-world environment, and then deployed to control a real mechanical agent in the particular real-world environment. Thus, in such cases, observations of the simulated environment relate to the real-world environment, and selected actions in the simulated environment relate to actions to be performed by the mechanical agent in the real-world environment.

[0163] Optionally, in any of the above implementations, the observations at any given time step may include data from previous time steps that may be useful in characterizing the environment, e.g., actions performed at the previous time step, rewards received at the previous time step, or both.

[0164] This specification uses the term "configured" in reference to systems and computer program components. One or more computer systems configured to perform a particular operation or behavior means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operation or behavior. One or more computer programs configured to perform a particular operation or behavior means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or behavior.

[0165] Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware containing the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The computer-readable storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver apparatus for execution by the data processing apparatus.

[0166] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may be or further include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0167] A computer program, which may also be referred to or written as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple cooperating files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program may be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0168] The term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and running on the same one or more computers.

[0169] The processes and logic flows described herein may be implemented by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data to generate output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0170] A computer suitable for running a computer program may be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Moreover, a computer may be incorporated in another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name a few.

[0171] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0172] To achieve interaction with a user, embodiments of the subject matter described herein may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, a keyboard, and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to achieve interaction with a user. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, speech input, or tactile input. Additionally, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from that web browser. A computer may also interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving reply messages from the user in response thereto.

[0173] A data processing apparatus for implementing machine learning models may also include, for example, dedicated hardware accelerator units for handling the typical computationally intensive part of machine learning training or production, i.e., inference, workloads.

[0174] The machine learning model may be implemented and deployed using a machine learning framework, for example, the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0175] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0176] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to a user device, e.g., for the purpose of displaying data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., as a result of user interaction, may be received at the server from the device.

[0177] While this specification contains details of many specific implementations, these should not be construed as limitations on the scope of the invention or the scope of what may be claimed, but rather as descriptions of features that may be unique to particular embodiments of a particular invention. Some features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, while features may be described above as operating in some combination, and may even be initially claimed as such, one or more features from a claimed combination may in some cases be omitted from the combination, and a claimed combination may be directed to a subcombination or a variation of a subcombination.

[0178] Similarly, while operations may be illustrated in the figures or described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in sequential order, or that all of the shown operations be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0179] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0180] Aspects of the present disclosure may be as set out in the following clauses. Clause 1. A method implemented by one or more computers, comprising: receiving a task description, the task description being a natural language description of a task to be performed by an agent in the environment; controlling the agent over a series of time steps to cause the agent to perform a task; the task comprising, at each of a plurality of time steps in the sequence of time steps: obtaining a first observation characterizing a state of the environment at a time step; processing a reporter input comprising the first observation using a reporter neural network to generate a reporter output defining a natural language report characterizing the agent's progress toward completing the task as of the time step; processing planner input comprising a task description and a natural language report using a planner neural network to generate natural language instructions for the agent; obtaining a second observation characterizing a state of the environment at the time step; processing the natural language instruction and the second observation using a policy neural network to generate a policy output that defines an action to be taken by the agent; selecting an action using the policy output; causing the agent to perform the selected action; A method comprising: Clause 2. The method of any preceding clause, wherein the planner neural network is pre-trained through unsupervised learning for the purpose of language modeling. Clause 3. Planner Input: an example task description; one or more exemplary reporter inputs; for each reporter input, an exemplary natural language report generated in response to the reporter input; 3. The method of clause 2, further comprising one or more prompt inputs, each comprising: Clause 4. The method of clauses 2 or 3, wherein the reporter output is a natural language report. Clause 5. The method of any one of clauses 2 to 4, wherein the reporter output comprises classification output across multiple categories. Article 6. selecting a category from a plurality of categories using the classification output; generating a natural language report, the natural language report including inserting natural language descriptions of selected categories at predetermined locations within the natural language report; 5. The method of clause 5, further comprising: Article 7. receiving a reward for the task in response to the agent performing the selected action; training a reporter neural network through reinforcement learning using the received rewards; The method of any preceding clause further comprising: Clause 8. The step of training a reporter neural network through reinforcement learning using the received rewards comprises: 8. The method of clause 7, comprising training the reporter neural network without training the policy neural network. Clause 9. The method of clause 8, wherein the policy neural network is trained on a task different from the task described by the task description. Clause 10. The method of any one of clauses 7 to 9, wherein the reporter neural network comprises one or more encoder neural networks that have been pre-trained. Clause 11. The method of clause 10, wherein the first observation comprises an image of the environment, and the reporter neural network comprises a first encoder neural network configured to process the image and pre-trained on a visual representation learning task. Clause 12. The method of clause 10 or 11, wherein the reporter input further comprises natural language instructions from a preceding time step, and the reporter neural network comprises a second encoder neural network configured to process the natural language instructions and pre-trained on a text representation learning task. Clause 13. The reporter neural network comprises an output sub-network configured to receive a respective output from each of the encoder neural networks and generate a reporter output from the respective outputs, and the step of training the reporter neural network comprises: training the output sub-network through reinforcement learning while keeping the pre-trained encoder fixed; or training the output subnetwork and the pre-trained encoder through reinforcement learning; 13. The method of any one of clauses 10 to 12, comprising: Clause 14. The method of any preceding clause where the first observation is the same as the second observation. Clause 15. The method of any preceding clause, in which the agent is a mechanical agent and the environment is a real-world environment. Clause 16. The method of clause 15, wherein the mechanical agent is a robot. Clause 17. The method of clause 15 or clause 16, wherein the first observation and the second observation comprise data generated from sensor measurements acquired by one or more sensors of the mechanical agent. Clause 18. The step of obtaining a task description comprises: The method of any preceding clause, comprising obtaining a task description as text input or speech input from a user. Clause 19. The method of any preceding clause, wherein the planner input further comprises one or more natural language reports from one or more preceding time steps in the series of time steps. Clause 20. Obtaining demonstration data, the demonstration data comprising natural language descriptions of example tasks and data characterizing performance of the example tasks by the expert agent; training a reporter neural network through imitation learning on demonstration data; The method of any preceding clause further comprising: Clause 21. The step of training a reporter neural network through imitation learning on demonstration data comprises: 21. The method of clause 20, comprising training the reporter neural network on demonstration data while holding the policy neural network and the planner neural network fixed. Clause 22. The method of any preceding clause, wherein the environment is a real-world environment of a service facility comprising a plurality of electronic devices, and the agent is an electronic agent configured to control the operation of the service facility. Clause 23. The method of any preceding clause, wherein the environment is a real-world manufacturing environment for producing a product, and the agent comprises an electronic agent configured to control a manufacturing unit or machine that operates to produce the product. Article 24. one or more computers; one or more storage devices communicatively coupled to one or more computers; wherein one or more storage devices store instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method operations of any one of clauses 1 to 23. Clause 25. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method acts of any one of clauses 1 to 23. [Explanation of symbols]

[0181] 100 Action Selection System 102 Action Selection Neural Network System 104 Agent 106 Environment 108 Action 110 Observations 112 Task Description 120 Policy Neural Networks 122 Policy Output 130 Planner Neural Network 132 Command 140 Reporter Neural Network 142 Report 150 Reward 410 Doer

Claims

1. A method implemented by one or more computers, comprising: receiving a task description, the task description being a natural language description of a task to be performed by an agent in an environment; controlling the agent over a series of time steps to cause the agent to perform the task; and wherein the task performs, at each of a plurality of time steps in the sequence of time steps: obtaining a first observation characterizing a state of the environment at the time step; processing a reporter input comprising the first observation using a reporter neural network to generate a reporter output defining a natural language report characterizing the agent's progress toward completing the task as of the time step; processing planner input comprising the task description and the natural language report using a planner neural network to generate natural language instructions for the agent; obtaining a second observation characterizing a state of the environment at the time step; processing the natural language instruction and the second observation using a policy neural network to generate a policy output that defines an action to be taken by the agent; selecting an action using the policy output; causing the agent to perform the selected behavior; A method comprising:

2. The method of claim 1 , wherein the planner neural network is pre-trained through unsupervised learning for the purpose of language modeling.

3. The planner input: an exemplary task description; one or more exemplary reporter inputs; for each reporter input, an exemplary natural language report generated in response to said reporter input; The method of claim 2 , further comprising one or more prompt inputs, each comprising:

4. 4. The method of claim 2 or 3, wherein the reporter output is the natural language report.

5. 5. The method of claim 2, wherein the reporter output comprises a classification output across multiple categories.

6. selecting a category from the plurality of categories using the classification output; generating the natural language report, the natural language report including inserting a natural language description of the selected category at a predetermined location within the natural language report; The method of claim 5 further comprising:

7. receiving a reward for the task in response to the agent performing the selected action; training the reporter neural network through reinforcement learning using the received rewards; The method of any of claims 1 to 6, further comprising:

8. training the reporter neural network through reinforcement learning using the received rewards; 8. The method of claim 7, comprising training the reporter neural network without training the policy neural network.

9. The method of claim 8 , wherein the policy neural network is trained on a different task than the task described by the task description.

10. 10. The method of claim 7, wherein the reporter neural network comprises one or more pre-trained encoder neural networks.

11. 11. The method of claim 10, wherein the first observation comprises an image of the environment, and the reporter neural network comprises a first encoder neural network configured to process the image and pre-trained on a visual representation learning task.

12. 12. The method of claim 10 or 11, wherein the reporter input further comprises natural language instructions from a previous time step, and the reporter neural network comprises a second encoder neural network configured to process the natural language instructions and pre-trained on a text representation learning task.

13. the reporter neural network comprises an output sub-network configured to receive a respective output from each of the encoder neural networks and generate the reporter output from the respective outputs, and training the reporter neural network comprises: training the output sub-network through reinforcement learning while keeping the pre-trained encoder fixed; or training the output sub-network and the pre-trained encoder through reinforcement learning; 13. The method of any one of claims 10 to 12, comprising:

14. 14. The method of claim 1, wherein the first observation is the same as the second observation.

15. The method of any of claims 1 to 14, wherein the agent is a mechanical agent and the environment is a real-world environment.

16. The method of claim 15 , wherein the mechanical agent is a robot.

17. 17. The method of claim 15 or 16, wherein the first observation and the second observation comprise data generated from sensor measurements acquired by one or more sensors of the mechanical agent.

18. obtaining the task description, 18. The method of any preceding claim, comprising obtaining the task description as text or speech input from a user.

19. 19. The method of claim 1, wherein the planner input further comprises one or more natural language reports from one or more previous time steps in the series of time steps.

20. acquiring demonstration data, the demonstration data comprising natural language descriptions of example tasks and data characterizing performance of the example tasks by an expert agent; training the reporter neural network through imitation learning on the demonstration data; 20. The method of claim 1, further comprising:

21. training the reporter neural network through imitation learning on the demonstration data; 21. The method of claim 20, comprising training the reporter neural network on the demonstration data while holding the policy neural network and the planner neural network fixed.

22. 22. The method of claim 1, wherein the environment is a real-world environment of a service facility comprising a plurality of electronic devices, and the agent is an electronic agent configured to control the operation of the service facility.

23. 23. The method of any of claims 1 to 22, wherein the environment is a real-world manufacturing environment for manufacturing a product, and the agent comprises an electronic agent configured to control a manufacturing unit or machine operating to manufacture the product.

24. one or more computers; and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective methods of any one of claims 1 to 23.

25. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method operations of any one of claims 1 to 23.

Images