Controlling agents using reporter neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling agents using a reporter neural network.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 410,156, filed on September 26, 2022. The disclosure of the prior application is considered a part of and incorporated by reference into the disclosure of the present application. Technical Field

[0003] This specification relates to processing data using machine learning models. Background Art

[0004] A machine learning model receives input and generates output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate output based on the received input and the values ​​of the model parameters.

[0005] Some machine learning models are deep models that employ multiple layers of the model to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. Summary of the invention

[0006] This specification generally describes a system implemented as a computer program on one or more computers at one or more locations that controls an agent that interacts with an environment to perform tasks in the environment.

[0007] Specifically, the system uses an action selection neural network system to control the agent, and the action selection neural network system includes a policy neural network, a planner neural network and a reporter neural network.

[0008] The present specification also describes training an action selection neural network system, and in particular a reporter neural network.

[0009] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0010] This specification describes controlling an agent using a planner neural network that generates natural language instructions and a policy neural network that receives the natural language instructions and the current observation and generates outputs that define actions to be performed by the agent.

[0011] For example, the planner neural network can be a pre-trained language model neural network, such as a Large Scale Language Model Neural Network (LSLM).

[0012] In a variety of different environments, such as in concretely implemented environments, logical reasoning is often required to accomplish complex tasks. Such logical reasoning, for example about the relationships between objects in the environment, about the properties of the environment or objects in the environment, or about the effects of the agent's actions on the environment, has always been a challenge for machine learning. Logical reasoning becomes even more challenging for concretely implemented agents, where the agent must perceive and control objects in its environment or itself in addition to reasoning about how to accomplish complex tasks.

[0013] However, recent language models have shown great potential for reasoning. However, these models are not concrete or grounded. They have no way to directly take actions in a concrete environment, or know what is happening in the environment. Therefore, directly using a language model neural network as a planner neural network has proven difficult.

[0014] To address these issues, the described techniques utilize a reporter neural network that is capable of generating a natural language report characterizing the agent's progress in performing a task, and providing the natural language report as part of the input to the planner neural network. By providing the planner neural network with access to this information, the system is able to effectively provide the planner neural network with the context necessary to generate effective instructions, thereby improving how well the policy neural network is able to control the agent to perform the task. In other words, by incorporating a reporter neural network into the scheme for controlling the agent, the system improves how well the agent performs various tasks, particularly where the task requires reasoning about aspects of the environment.

[0015] More specifically, because the planner neural network does not have the ability to interrogate the environment to observe the effects of actions performed by the agent, the natural language reports generated by the reporter neural network provide relevant information about the environment that the planner neural network is able to use to adjust instructions issued to the policy neural network (and therefore, instructions issued to the agent), and that would otherwise be unavailable to the planner neural network.

[0016] Additionally, when the planner neural network is a pre-trained language model and the observations received are images of the environment or other sensor readings of the environment, the planner and policy networks do not operate on the same input space: the policy network receives observations, such as images or other sensor readings, and produces policy outputs that define actions, while the planner neural network receives natural language inputs (task descriptions and natural language reports) and produces natural language outputs (generated natural language instructions). While the policy network is language-conditional and is able to interpret the planner's instructions, the planner cannot parse the consequences of the actor's actions (to produce an appropriate next action) because the planner operates only on textual inputs and not on observations that, for example, characterize the state of the environment. To bridge this gap, the reporter neural network converts the agent's action and observation space into the planner's input space by using the natural language reports generated by the reporter neural network.

[0017] Additionally, the specification also describes techniques for training a reporter neural network, for example, through reinforcement learning or imitation learning or both. Specifically, the system is able to train a reporter neural network to learn which information is most helpful to the planner, and eventually converge to only report true and relevant information to the planner at any given time step. This eliminates the need to specify in advance which information will be relevant to the planner for any given task, and allows the system to adapt and generalize to new tasks that require different types of information to be provided to the planner in order for the planner to effectively control the agent.

[0018] In general, by incorporating a reporter neural network and a planner neural network into a loop along with a policy neural network, the resulting system is able to effectively perform any of a variety of tasks that require a combination of skills, such as two or more of the following examples. For example, the system is able to effectively perform tasks that require logical reasoning: the ability to take complex instructions and perform different kinds of logical operations on them to determine the correct course of action. As another example, the system is able to effectively perform tasks that require generalization: the ability to generalize beyond the agent's prior experience. As another example, the system is able to effectively perform tasks that require exploration: the ability to explore the world around the agent to reveal new information that can inform its reasoning about what actions to take. As another example, the system is able to effectively perform tasks that require perception: the ability to use the raw observations that the agent has (usually vision) and process the world and make decisions using what it sees.

[0019] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 An example action selection system is shown.

[0021] Figure 2 is a flowchart of an example process for controlling an agent at a given time step.

[0022] Figure 3 is a flow chart of an example process for training a reporter neural network via reinforcement learning.

[0023] Figure 4 An example of the operation of an action selection neural network system is shown.

[0024] Figure 5 An example of improved agent control achieved by combining a planner neural network and a reporter that acts as a bridge between the policy neural network and the planner neural network is shown.

[0025] Like reference numbers and names in the various drawings indicate like elements. DETAILED DESCRIPTION

[0026] Figure 1 An example action selection system 100 is shown. Action selection system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0027] The action selection system 100 uses an action selection neural network system 102 to control an agent 104 interacting with an environment 106 to complete a task by selecting an action 108 to be performed by the agent 104 at each of a plurality of time steps during an episode of performing the task.

[0028] Examples of agents, environments, and tasks are described below.

[0029] An "episode" of a task is a sequence of interactions during which the agent attempts to perform an instance of the task starting from some starting state of the environment. In other words, each task episode begins with the environment in an initial state - e.g., a fixed initial state or a randomly selected initial state - and ends when the agent has successfully completed the task or when some termination criterion is met, e.g., when the environment enters a state that has been designated as a terminal state or when the agent has performed a threshold number of actions without successfully completing the task.

[0030] Prior to performing a task round, the system 100 receives a task description 112 that defines the task to be performed during the task round. Specifically, the task description 112 is a natural language description of the task to be performed by the agent 104 in the environment 106. For example, the system 100 can receive the task description 112 from the user as text input or audio input (which the system then converts to text).

[0031] Thus, an additional constraint on a task round can be that the round continues over a sequence of time steps until one of the above criteria is met or until a new task description is received that specifies a new task to be performed (or modifies an existing task).

[0032] At each time step during any given mission episode, the system 100 receives an observation 110 characterizing the current state of the environment 106 at that time step, and in response, selects an action 108 to be performed by the agent 104 at that time step.

[0033] Observations 110 can include any suitable information characterizing the state of the environment. As an example, observations 110 can include sensor readings from one or more sensors configured to sense the environment. For example, observations 110 can include one or more images captured by one or more cameras, measurements from one or more proprioceptive sensors, and the like.

[0034] After the agent performs action 108, the environment 106 transitions to a new state.

[0035] In some cases, the system 100 receives a reward 150 from the environment in response to the agent performing an action.

[0036] Typically, rewards are scalar values ​​and represent the agent's progress toward completing the task.

[0037] As a specific example, the reward can be a sparse binary reward that is zero unless the task is successfully completed, and is one if the task is successfully completed due to the action performed.

[0038] As another specific example, the reward can be a dense reward that measures the agent's progress toward completing the task based on individual observations received during episodes of attempting to perform the task, i.e., such that a non-zero reward can be received and is received frequently before successfully completing the task.

[0039] When executing any given mission round, the system 100 selects actions in an attempt to maximize the rewards received during the mission round.

[0040] That is, at each time step during an episode, the system 100 selects an action that attempts to maximize the reward that will be received for the remainder of the task episode starting at that time step.

[0041] In general, at any given time step, the reward you will receive is a combination of the rewards you will receive at the time steps after the given time step in the episode.

[0042] For example, at time step t, the reward can satisfy:

[0043] ∑ i γ i-t-1 r i

[0044] where the range of i is all time steps after t in the episode, or some fixed number of time steps after t in the episode, γ is the discount factor, and r i is the reward at time step i.

[0045] As described above, system 100 uses action selection neural network system 102 to control agent 104.

[0046] Action selection system 102 includes a policy neural network 120 , a planner neural network 130 , and a reporter neural network 140 .

[0047] The policy neural network 120 is a neural network configured to process observations and natural language instructions 132 as input to generate a policy output 122 that defines an action to be performed by the agent. The system 100 then uses the policy output 122 to select an action 108 to be performed by the agent 104.

[0048] Policy neural network 120 can have any suitable architecture that allows policy neural network 120 to map observations and natural language instructions to policy outputs.

[0049] For example, the policy neural network 120 can include one encoder for observations, another encoder for natural language instructions, and a policy subnetwork configured to process the outputs from the two encoders to generate a policy output. In some cases, the policy subnetwork can include a memory neural network, such as a recurrent neural network or a Transformer neural network, so that the policy output 122 at a given time step incorporates information from previous time steps.

[0050] As a specific example, when the neural network 120 receives a visual observation, such as an image or video, the neural network can have a convolutional visual encoder to encode the visual observation and a recurrent neural network, such as an LSTM-based language encoder to encode the action instructions. The neural network can also have an LSTM-based memory module to help take previous actions and observations into account for policy output.

[0051] In one example, the policy output 122 can include a corresponding Q-value for each action in the fixed set. The system 100 can process the Q-values ​​(e.g., using a soft-max function) to generate a corresponding probability value for each action, which can be used to select an action or can select the action with the highest Q-value.

[0052] The Q-value of an action is an estimate of the “reward” that the agent would receive from performing the action in response to the current observation and then being controlled using the action generated by the action selection system.

[0053] In another example, the policy output 122 may include a corresponding numerical probability value for each action in the fixed set. The system 100 can select an action, for example, by sampling the actions according to the probability values, by selecting the action with the highest probability value.

[0054] As another example, when the action space is continuous, the policy output 122 can include parameters of a probability distribution over the continuous action space. The system 100 can then select an action by sampling actions from the probability distribution or by selecting an average action.

[0055] Planner neural network 130 is a neural network configured to process planner input to generate natural language instructions 132 as output for policy neural network 120 .

[0056] Typically, the planner input includes a task description 112 and a natural language report 142, which, as described in more detail below, describes the current state of the task round. That is, the natural language report 142 at a given time step during the task round characterizes the progress of the agent in completing the task as of the given time step.

[0057] The natural language instructions can be considered as “planning” outputs that provide policy neural network 122 with high-level information that is useful in performing a task, i.e., they guide policy neural network 120 to select appropriate “low-level” actions to control the agent.

[0058] For example, planner neural network 130 can be a language model neural network that has been trained on a language modeling objective. That is, the language model neural network can have been pre-trained on a language modeling objective on a large corpus of training data. In some embodiments, planner neural network 130 has not yet been trained for controlling an agent, i.e., has not yet been jointly trained with policy neural network 120 or reporter neural network 140.

[0059] A language model neural network is a neural network configured to process an input to generate an output comprising a probability distribution over a set of text tokens in a vocabulary of text tokens, where the probability of each token represents the likelihood that the text token immediately follows the input.

[0060] For example, the language model neural network can be an autoregressive language model neural network.

[0061] Language model neural networks are called autoregressive neural networks because the neural network autoregressively generates an output sequence of tokens by generating each particular token in the output sequence conditioned on a current input sequence, including any tokens that precede the particular text token in the output sequence, i.e., tokens that have been generated for any previous positions in the output sequence that precede the particular position of the particular token, and context inputs ("context sequence") that provide context for the output sequence.

[0062] For example, when generating a word-gram at any given position in the output sequence, the current input sequence can include a context sequence and a word-gram at any previous position before the given position in the output sequence. As a specific example, the current input sequence can include a context sequence followed by a word-gram at any previous position before the given position in the output sequence. Optionally, the context sequence and the current output sequence can be separated by one or more predetermined word-grams within the current input sequence.

[0063] More specifically, to generate a specific word-gram at a specific position within the candidate output sequence, the neural network can process the current input sequence to generate a score distribution, e.g., a probability distribution, which assigns a corresponding score, e.g., a corresponding probability, to each text word-gram in the vocabulary of text words. The neural network 110 can then use the score distribution to select a text word-gram from the vocabulary as the specific word-gram. For example, the neural network can greedily select the word-gram with the highest score, or can sample a word-gram from the distribution, e.g., using kernel sampling or another sampling technique.

[0064] As a specific example, the language model neural network can be a Transformer-based autoregressive neural network that includes (i) multiple attention blocks, each of which applies a self-attention operation, and (ii) an output subnetwork that processes the output of the last attention block to generate a score distribution.

[0065] The neural network can have any of a variety of Transformer-based neural network architectures. Examples of such architectures include those described in: J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, DdL Casas, LA Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language model, arXiv preprint arXiv:2203.15556, 2022; J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche,LAHendricks,M.Rauh,P.Huang,A.Glaese,J.Welbl,S.Dathathri,S.Huang,J.Uesato,J.Mellor,I.Higg ins,A.Creswell,N.McAleese,A.Wu,E.Elsen,SMJayakumar,E.Buchatskaya,D.Budden,E.Sutherland,K.Simonyan, M.Paganini,L.Sifre,L.Martens,XLLi,A.Kuncoro,A.Nemazadeh,E.Gribovskaya,D.Donato,A.Lazaridou,A.Mens ch,J.Lespiau,M.Tsimpoukelli,N.Grigorev,D.Fritz,T.Sottiaux,M.Pajarskas,T.Pohlen,Z.Gong,D.Toyama,C.de Masson d'Autume,Y.Li,T.Terzi,V.Mikulik,I.Babuschkin,A.Clark,D.de Las Casas,A.Guy,C.Jones,J.Bradbury,M.Johnson,BAHechtman,L.Weidinger,I.Gabriel,WSIsaac,E.Lockhart,S.Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. .

[0066] However, typically, a Transformer-based neural network includes a sequence of attention blocks, and during processing of a given input sequence, each attention block in the sequence receives a corresponding input hidden state for each input token in the given input sequence. The attention block then updates the hidden state of at least the last token in the given input sequence at least in part by applying self-attention to generate a corresponding output hidden state for the last token. The input hidden state of the first attention block is an embedding of the input token in the input sequence, and the input hidden state of each subsequent attention block is an output hidden state generated by the previous attention block.

[0067] In this example, the output subnetwork processes the output hidden state generated by the last attention block in the sequence for the last input token in the input sequence to generate a score distribution.

[0068] As described above, system 100 or another training system pre-trains the language model neural network on a language modeling task, e.g., a task requiring predicting the next token following the current sequence in the training data given a current sequence of text tokens, prior to using the language model neural network as planner neural network 130. As a specific example, the language model neural network can be pre-trained on a maximum likelihood objective on a large dataset including text, such as text publicly available from the Internet or another text corpus.

[0069] For example, reporter neural network 130 can be a pre-trained large-scale language model (LSLM). For example, LSLM can be one of two variants of the Chinchilla model described in the above-linked Hoffmann et al. paper: one with 70 billion parameters (referred to as the 70B model) and the other with 7 billion parameters (referred to as the 7B model).

[0070] In this example, after the first time step in a task round, the planner input can also include the reporter input of one or more previous time steps and the natural language instructions generated at one or more previous time steps. Including these previous outputs can provide context for the planner neural network 130 to generate appropriate natural language instructions at the current time step.

[0071] In this example, the planner input can also include one or more prompt inputs that provide the planner with access to examples of "ground truth" agent controls in response to the example task description. That is, the planner input can include k-trial prompts, where k is an integer greater than or equal to one, which includes k prompt inputs. For example, each planner input can include: an example task description; one or more example reporter inputs; and for each reporter input, an example natural language report generated in response to the reporter input. Including k-trial prompts can assist the planner neural network 130 in performing "context learning" to adapt to generate appropriate natural language instructions.

[0072] Reporter neural network 140 is a neural network configured to process reporter input comprising observations to generate reporter output defining a natural language report 142, i.e., a report 142 characterizing the agent's progress in completing the task as of time step.

[0073] In some implementations, the reporter output is a natural language report 142. That is, the reporter neural network 140 is a generative neural network that processes observations to generate a natural language output.

[0074] For example, the reporter neural network 140 can have any suitable visual language model (VLM) neural network architecture. A visual language model (VLM) is a neural network that receives an image and optionally text as input, and generates text as output. For example, a VLM can include an encoder neural network, such as a visual Transformer or a convolutional neural network, that processes an image to generate an encoded representation of the image; and a decoder neural network, such as a Transformer-based decoder neural network, that generates text conditioned on the encoded representation of the image and optionally an embedding of the input text. An example of a VLM is the Flamingo model, which is described in Flamingo: a Visual Language Model for Few-Shot Learning, which is available at arXiv:2204.14198. Another example of a VLM is described in Multimodal Few-Shot Learning with Frozen Language Models, which is available at arXiv:2106.13884.

[0075] In some other embodiments, the reporter output can be a classification output of multiple categories, for example, multiple categories each represent different progress levels of completing the task. As an example, when the task is to navigate to a specified location, each category can represent the different distance ranges between the agent and the specified location. As another example, when the task is to locate an object with specified attributes, each category can represent the different attributes of the object closest to the agent in the environment. In these embodiments, the system 102 can generate a report 142 by selecting a category from multiple categories using the classification output, for example, by sampling from the classification output or by selecting the category with the highest score according to the classification output, and by inserting the natural language description of the selected category at the predetermined position in the natural language report to generate a natural language report.

[0076] In these embodiments, reporter neural network 140 can have any suitable architecture that allows a neural network to map observations to classification outputs, such as a convolutional neural network, a fully connected neural network, or a self-attention based neural network.

[0077] For example, when the observations include an image, the reporter neural network 140 can include a first encoder subnetwork, such as a visual Transformer or a convolutional neural network, that encodes the image.

[0078] In some embodiments, one or more of the encoder neural networks have been pre-trained, for example, as part of a classification neural network, or pre-trained on a representation learning task. As a specific example, when observing an image including an environment and the reporter neural network includes a first encoder neural network configured to process the image, the first encoder can have been pre-trained on a visual representation learning task.

[0079] As another example, when the reporter input also includes natural language instructions from one or more previous time steps (i.e., in addition to the observations), the reporter neural network can include a second encoder neural network configured to process the natural language instructions to encode the instructions. In some of these cases, the second encoder can have been pre-trained on a text representation learning task.

[0080] Using a pre-trained encoder can allow system 100 to leverage representations learned on large-scale data, which can result in accurate reports being generated by the reporter neural network.

[0081] In some implementations, policy neural network 120 and reporter neural network 140 each receive the same type of observations, ie, each receive observation 110 .

[0082] In some other embodiments, policy neural network 120 and reporter neural network 140 receive different types of observations that contain different information about the environment. For example, when observation 110 includes data generated by multiple sensors, policy neural network 120 can receive a second observation that includes data from a second subset of the sensors, while reporter neural network 140 receives a first observation that includes data from a first subset of the sensors.

[0083] In some embodiments, the reporter neural network 140 can include a memory subnetwork, such as a recurrent neural network or a Transformer neural network, so that the reporter neural network 140 can incorporate context from previous time steps.

[0084] In some implementations, the reporter input includes additional information in addition to the observation, such as actions performed at one or more previous time steps.

[0085] In general, at any given time step during a task round, the system 100 uses the reporter neural network 140 and the planner neural network 130 to generate inputs to the policy neural network 120, and then uses the policy output 122 generated by the policy neural network 120 to select an action 108 to be performed by the agent 104 at that time step.

[0086] The system 100 then causes the agent 104 to perform the selected action 108 .

[0087] The following will refer to Figure 2 This is described in more detail.

[0088] Figure 2 is a flow chart of an example process 200 for controlling an agent at a given time step during a task round. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to the present specification—e.g. Figure 1 Action selection system 100 - capable of performing process 200.

[0089] The system obtains a first observation characterizing a state of the environment at a time step (step 202).

[0090] The system processes a reporter input including a first observation using a reporter neural network to generate a reporter output (step 204).

[0091] Typically, a reporter output defines a natural language report that characterizes the agent's progress in completing the task as of time step .

[0092] In some cases, the reporter input can also include additional information, such as a task description, natural language instructions provided to the agent at previous time steps, actions performed at one or more previous time steps, etc.

[0093] The system processes the planner input including the task description and the natural language report using the planner neural network to generate natural language instructions for the agent (step 206).

[0094] The system obtains a second observation characterizing the state of the environment at the time step (step 208).

[0095] In some cases, the first observation and the second observation are the same. For example, both can include information generated by sensor readings generated by the same set of one or more sensors sensing the environment at the time step.

[0096] In some other cases, the first observation and the second observation can include different information. For example, the first observation and the second observation can include information generated by sensor readings generated by different combinations of sensors sensing the environment at the time step.

[0097] The system processes the natural language instruction and the second observation using a policy neural network to generate a policy output (step 210). As described above, the policy output defines the action to be performed by the agent.

[0098] The system controls the agent using the policy output, ie, by selecting an action using the policy output and causing the agent to perform the selected action (step 212).

[0099] As described above, as the agent performs the chosen action, the environment typically changes state, and the system receives the next observation at the next time step, which characterizes the new state of the environment.

[0100] The system is then able to use the next observation and the reporter neural network to generate a new natural language report for the next time step.

[0101] Optionally, the system is then able to receive a reward indicating the agent's progress in completing the task.

[0102] Therefore, because the planner neural network does not have the ability to interrogate the environment to observe the effects of actions performed by the agent, the natural language reports generated by the reporter neural network provide relevant information about the environment that the planner neural network can use to adjust the instructions issued to the policy neural network (and therefore to the agent).

[0103] Additionally, when the planner neural network is a pre-trained language model and the received observations are images of the environment or other sensor readings of the environment, the planner and policy network do not operate on the same observation space: the policy network operates on pixel observations (or other sensor readings) and produces actions, while the planner operates on language observations (cues) and produces language actions (generated instructions).

[0104] While the policy network is language-conditional and able to interpret the planner’s instructions, the planner cannot parse the consequences of the actor’s actions (to produce an appropriate next action). Therefore, the reporter translates from the actor’s action and observation space to that of the planner.

[0105] In some embodiments, the planner and reporter execute less than all time steps in a given task round. For example, the planner and reporter can execute at every n time steps, where n is a fixed integer greater than one. In these embodiments, the policy neural network can continue to receive the same natural language instructions at different time steps until a new natural language instruction is generated by the planner.

[0106] When the system receives a reward in response to performing an action, the system is able to use the received reward to train a reporter neural network through reinforcement learning, i.e., to maximize the expected return received in response to the action performed by the agent.

[0107] This is Figure 3 Described in.

[0108] Figure 3 is a flow chart of an example process 300 for training a reporter neural network. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to the present specification, such as Figure 1 The action selection system 100 can perform the process 300.

[0109] The system control agent uses the reporter neural network, the policy neural network and the planner neural network to perform actions (step 302), such as described above with reference to Figure 2 described.

[0110] The system receives a reward for the agent performing the action (step 304).

[0111] The system trains the reporter neural network using rewards through reinforcement learning (step 306).

[0112] For example, the system can store a transformation including an observation, a task description, a natural language report, and a reward in a replay memory. The system can then repeatedly sample a transformation or a sequence of transformations from the replay memory and use the sampled transformations to train a reporter neural network through reinforcement learning.

[0113] The system can use any suitable reinforcement learning training technique to train the reporter neural network. Examples of reinforcement learning techniques that can be used include V-Trace, PPO, MPO, etc.

[0114] As a specific example, the system is able to train a reporter neural network while keeping the planner neural network and the policy neural network fixed. In this example, although rewards are received as a result of controlling the agent using the policy output, the system does not use the rewards to train the policy neural network, but instead only uses the rewards to train the reporter neural network, which generates natural language reports that "indirectly" select actions (and therefore, rewards) only by changing the way the planner neural network is adjusted.

[0115] Specifically, the planner neural network and the policy neural network can have been pre-trained on the respective objectives, and the system can keep the parameters of the planner neural network and the policy neural network fixed to values ​​generated as a result of this pre-training.

[0116] For example, the policy neural network can have been trained on one or more tasks different from the one specified by the task description through reinforcement learning or imitation learning. For example, the policy neural network can have been trained using a task description generated by a user or by a different planner neural network. Additionally, as described above, the planner neural network can have been pre-trained on a language modeling task and can remain fixed during reporter training (and not used during policy neural network training).

[0117] Thus, as reinforcement learning training of the reporter neural network continues, the system is able to use rewards—such as the final reward of an episode for a sparse reward task—to reinforcement learn which information is most helpful to the planner neural network, where the training of the reporter eventually converges to only reporting true and relevant information to the planner at any given time step. This eliminates the need to specify in advance what information will be relevant to the planner for any given task, and allows the system to adapt and generalize to new tasks that require different types of information to be provided to the planner in order for it to effectively control the agent.

[0118] In some cases, instead of or in addition to training the reporter neural network through reinforcement learning, the system can also have access to a set of demonstration data, which includes sequences of observation-action pairs that characterize an expert agent interacting with an environment to perform corresponding tasks, and for each corresponding task, a natural language description of the task. The system can then train the reporter neural network on this demonstration data through imitation learning, such as through behavioral cloning or adversarial imitation learning.

[0119] Additionally, as described above, in some embodiments, the reporter neural network can have one or more encoders that have been pre-trained on other tasks. In these embodiments, the system can (i) keep these encoders fixed during the training of the reporter neural network described above, and only train the rest of the reporter neural network, or (ii) fine-tune the pre-trained encoders together with the rest of the reporter neural network.

[0120] Figure 4 An example 400 of the operation of the system in controlling an agent to perform a task round is shown.

[0121] exist Figure 4 In a simplified example of , the task requires information gathering. The goal of the episode is to pick up the correct object based on a secret property of another object. The task passed to the planner neural network is described as follows: "if {decider object} is good, then pick up {object 1}, otherwise pick up {object 2}".

[0122] To successfully complete an episode of this task, five high-level steps need to be performed: a) the planner instructs the actor (Actor) to inspect {decider object}, b) the actor inspects the object, c) the reporter conveys the revealed information, d) the planner infers which object needs to be picked up based on the report, {object 1} or {object 2}, and instructs the actor to pick up the correct object e) the actor picks up the correct object.

[0123] exist Figure 4 In the example of , the "solid blue circle" is the decider object, the checkered green triangle is object 1, and the striped yellow cross is object 2. Therefore, in Figure 4 In the example of , the task description 402 of the round is "if the solid blue circle is good, pick up the checkered green triangle, otherwise pick up the striped yellow cross."

[0124] To control the agent, actor 410 implements an instance of a policy neural network and uses the policy neural network to take actions in the environment in response to received observations. At a given time step, the system uses reporter neural network 140 to generate a natural language report 420 that says "I checked the solid blue circle and it has a good secret property."

[0125] The system provides this report 420 along with the task description 402 to the planner neural network 130, which generates natural language instructions 430 for the agent to "check the blue circle."

[0126] The system provides natural language instructions 430 to actor 410 , which uses instructions 430 and observations to generate input to policy neural network 120 .

[0127] Thus, the report 420 provides information to the planner neural network 130 about the progress of the agent in performing the task, for example, about which objects the agent has inspected so far and what their properties are. The planner neural network 130 can then use this information to update the instructions provided to the agent so that the agent inspects the relevant objects in order to successfully complete the task.

[0128] Figure 5 An example 500 of improvements in agent control achieved by combining a planner neural network and a reporter that acts as a bridge between the policy neural network and the planner neural network is shown.

[0129] Specifically, Figure 5 Part A of FIG. 5 shows the performance of the information gathering task utilizing: a) a 7B parameter planner neural network, b) a 70B parameter planner neural network, and c) a “trained RL baseline” that does not use a reporter neural network or a planner neural network, but instead directly provides “reports” to the policy neural network similar to those generated by the reporter neural network.

[0130] Specifically, Figure 5 The example shows the performance of a "conditional" information-gathering task - such as the one described above - and a "search" information-gathering task. The "search" task extends the "conditional" task by requiring an additional information-gathering step. Instead of inspecting a single object, the agent needs to inspect multiple objects, annotate their secret properties, and pick the correct object to get a reward. For example, the task description could be specified as "The objects are {}, {}, {}, and {}. Pick the object with the good secret property." A successful round consists of the planner asking the actor to inspect each object in turn until it finds an object with a "good" property, at which point it asks the actor to pick up that object.

[0131] from Figure 5 As can be seen in Section A of the , both the 7B parameter and 70B parameter planners outperform the “trained RL” baseline on both tasks, with the 70B parameter planner (and the corresponding increase in reasoning power encoded within the parameters of the larger planner neural network) significantly outperforming the baseline.

[0132] As can be seen from the baselines, both tasks are difficult for reinforcement learning baselines, even when the policy neural network is provided with “reports” as additional input.

[0133] In some cases, the reporter may provide “noisy” information to the planner. That is, due to the variability of the task, and because accurately describing the agent’s progress is a difficult task, the natural language reports may sometimes be wrong or irrelevant to the plan.

[0134] Figure 5 Part B of FIG. 4 shows the robustness of the 7B, 70B, and baseline variants to irrelevant reports, e.g., where reports are forced to describe irrelevant actions 20% of the time. As shown in Part B, both the 7B and 70B variants outperform the baseline in the presence of irrelevant reports. Furthermore, when the k-trial prompt planner describes how to handle irrelevant reports, performance rebounds to almost original levels (where irrelevant reports are reduced).

[0135] Part C shows the increase in training reward achieved when the reporter neural network is trained on the visual conditional task while keeping the planner and policy neural networks fixed. As can be seen from Part C, training the reporter neural network significantly improves the performance of the policy neural network in controlling the agent.

[0136] An example of a visual conditional task is one in which the target object changes depending on the color of the agent. The language model — i.e. the planner — does not have access to this agent color information. Therefore, the reporter must learn to decode it from the visual observations and report it back. The language model is then able to issue the correct next instruction, which results in a reward.

[0137] A more complex example of a visual conditional task that combines the challenges of multi-step tasks and learning reports is a task in which the agent must perform information gathering actions in order to obtain relevant information. For example, which object the agent must pick up can depend on whether the specified decision maker object is close to a wall or somewhere else in the environment. To determine this, the agent must navigate to the object to examine its surroundings.

[0138] As mentioned above, including a reporter neural network can improve the operation of the system in a number of ways. This is true even when the reporter provides noisy information to the planner.

[0139] Table 1 below shows some examples of this, where the output of the reporter is represented as an actor and the output of the planner is represented as a LM. The first box highlights the error caused by the reporter, and the second box shows how the planner causes the policy neural network and the planner to repeat the report, resulting in the error being corrected.

[0140]

[0141] Table 1

[0142] In some cases, the planner neural network can be made to repeat instructions, for example through k-trial prompts, in an attempt to get the reporter to provide relevant information instead of irrelevant or noisy information. This is shown in Table 2 below.

[0143] QUESTION: The objects in the room are noisy dark red tee, verticalstriped cyan plus, noisy purple u and horizontal striped pink triangle. Get the object with a good secret property.

[0144] ANSWER:

[0145] LM: Examine noisy dark red tee. <eos>

[0146] Agent:I have moved up and right. <eos>

[0147] LM:Examine noisy dark red tee. <eos>

[0148] Agent:I have moved up and right. <eos>

[0149] LM:Examine noisy dark red tee. <eos>

[0150] Agent:I have moved down and right. <eos>

[0151] LM:Examine noisy dark red tee. <eos>

[0152] Agent:I have moved down and left. <eos>

[0153] LM:Examine noisy dark red tee. <eos>

[0154] Agent:I examined noisy dark red tee.Its secret property has valuebad. <eos>

[0155] LM:Examine vertical striped cyan plus. <eos>

[0156] Agent:I have moved left. <eos>

[0157] LM:Examine vertical striped cyan plus. <eos>

[0158] Agent:I examined vertical striped cyan plus.Its secret property hasvalue good. <eos>

[0159] LM:Pickup vertical striped cyan plus. <eos>

[0160] Table 2

[0161] In some embodiments, the planner can loop through commands, for example, to check various objects until it receives a report of a good object, as shown in Table 3 below:

[0162] QUESTION: The objects in the room are grid pink tee, horizontal striped yellow h and horizontal striped yellow upside down tee, vertical striped light yellow h and noisy dark blue upside down u. Get the object with a good secret property.

[0163] ANSWER:

[0164] LM: Examine grid pink tee. <eos>

[0165] Agent:I have moved up. <eos>

[0166] LM:Examine horizontal striped yellow upside down tee. <eos>

[0167] Agent:I have moved up and right. <eos>

[0168] LM:Examine vertical striped light yellow h. <eos>

[0169] Agent:I have moved down and left. <eos>

[0170] LM:Examine noisy dark blue upside down u. <eos>

[0171] Agent:I examined noisy dark blue upside down u.Its secret propertyhas value bad. <eos>

[0172] LM:Examine grid pink tee. <eos>

[0173] Agent:I have moved down and left. <eos>

[0174] LM:Examine horizontal striped yellow upside down tee. <eos>

[0175] Agent:I have moved down. <eos>

[0176] LM:Examine vertical striped light yellow h. <eos>

[0177] Agent:I have moved down. <eos>

[0178] LM:Examine noisy dark blue upside down u. <eos>

[0179] Agent:I examined vertical striped light yellow h.Its secret propertyhas value bad. <eos>

[0180] LM:Examine grid pink tee. <eos>

[0181] Agent:I have moved up and right. <eos>

[0182] LM:Examine horizontal striped yellow upside down tee. <eos>

[0183] Agent:I examined grid pink tee.Its secret property has value good. <eos>

[0184] LM:Pickup grid pink tee. <eos>

[0185] Table 3

[0186] In some embodiments, the environment is a real-world environment, the agent is a mechanical agent that interacts with the real-world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle that operates in or navigates through the environment, and the actions are actions taken by the mechanical agent to perform tasks in the real-world environment. For example, the agent can be a robot that interacts with the environment to complete a specific task, such as locating an object of interest in the environment or moving an object of interest to a specified location in the environment or navigating to a specified destination in the environment.

[0187] In these embodiments, observations may include, for example, one or more of the following: images, object position data, and sensor data to capture observations when the agent interacts with the environment, such as sensor data from images, distance or position sensors, or from actuators. For example, in the case of a robot, observations may include data characterizing the current state of the robot, such as one or more of the following: joint positions, joint velocities, joint forces, torques, or accelerations, such as gravity-compensated torque feedback, and the global or relative posture of an item held by the robot. In the case of a robot or other mechanical agent or vehicle, observations may similarly include one or more of the position, linear or angular velocity, force, torque, or acceleration, and global or relative posture of one or more components of the agent. Observations may be defined as 1, 2, or 3 dimensions, and may be absolute and / or relative observations. Observations may also include, for example, sensed electronic signals, such as motor currents or temperature signals; and / or, for example, images or video data from a camera or LIDAR sensor, for example, data from a sensor of the agent or data from a sensor located separately from the agent in the environment.

[0188] In these embodiments, the action can be a control signal for controlling a robot or other mechanical agent, for example, a torque of a joint of a robot or a higher level control command, or a torque of an autonomous or semi-autonomous land, air, or sea vehicle, for example, a torque of a control surface or other control element, for example, a steering control element of a vehicle, or a higher level control command. The control signal can include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or a component of another mechanical agent. The control signal can also or instead include electronic control data, such as motor control data, or more generally include data for controlling one or more electronic devices within an environment, whose control has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, the control signal can define actions for controlling navigation, such as steering, and movement, such as braking and / or acceleration of a vehicle.

[0189] In some embodiments, the environment is a simulation of the real world environment described above, and the agent is implemented as one or more computers that interact with the simulated environment. For example, the simulated environment can be a simulation of a robot or a vehicle, and the reinforcement learning system can be trained on the simulation and then, once trained, used in the real world.

[0190] In some embodiments, the environment is a real-world manufacturing environment for manufacturing products such as chemical, biological or mechanical products or food. As used herein, "manufacturing" products also includes refining starting materials to create products, or processing starting materials, such as to remove pollutants, to generate clean or recycled products. A manufacturing plant may include multiple manufacturing units, such as containers for chemical or biological substances, or machines for processing solids or other materials, such as robots. The manufacturing units are configured so that intermediate versions or components of the product can be moved between manufacturing units during product manufacturing, for example, via pipelines or mechanical transmission. As used herein, product manufacturing also includes manufacturing food by kitchen robots.

[0191] The agent may include an electronic agent configured to control a manufacturing cell or a machine, such as a robot, that operates to manufacture a product. That is, the agent may include a control system configured to control the manufacture of a chemical, biological, or mechanical product. For example, the control system may be configured to control one or more of the manufacturing cells or machines, or to control the movement of intermediate versions or components of a product between manufacturing cells or machines.

[0192] As an example, the tasks performed by the agent may include the task of manufacturing a product or an intermediate version or component thereof. As another example, the tasks performed by the agent may include the task of controlling - e.g., minimizing - the use of resources, such as the task of controlling the consumption of electricity, or the consumption of water, or the consumption of any material or consumable used in the manufacturing process.

[0193] Actions may include control actions to control the use of a machine or manufacturing unit that processes solid or liquid materials to manufacture a product or an intermediate or component thereof, or to control the movement of intermediate versions or components of a product within a manufacturing environment, such as between manufacturing units or machines. In general, an action may be any action that has an effect on the observed state of the environment, such as an action configured to adjust any of the sensed parameters described below. These may include actions that adjust the physical or chemical conditions of a manufacturing unit, or actions that control the movement of mechanical parts of a machine or joints of a robot. These actions may include actions that impose operating conditions on a manufacturing unit or machine, or actions that cause settings to change to adjust, control, or turn on or off the operation of a manufacturing unit or machine.

[0194] The reward or reward may be related to a measure of the performance of the task. For example, in the case of a task to manufacture a product, the measure may include a measure of the number of products manufactured, the quality of the product, the speed at which the product is produced, or the physical cost of performing the manufacturing task, such as a measure of the amount of energy, material, or other resource used to perform the task. In the case of a task to control the use of a resource, the measure may include any measure of the use of the resource.

[0195] In general, observations of the state of the environment may include any electronic signals representing electronic and / or mechanical items of equipment. For example, representations of the state of the environment may be derived from observations of sensors that sense the state of the manufacturing environment, such as sensors that sense the state or configuration of a manufacturing cell or machine, or sensors that sense the movement of materials between manufacturing cells or machines. As some examples, such sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow / movement rate, or flow / movement path of one or more materials; physical or chemical conditions, such as physical state, shape, or configuration, or chemical state, such as pH; configuration of a cell or machine, such as a mechanical configuration of a cell or machine, or valve configuration; image or video sensors for capturing image or video observations of a manufacturing cell or machine or movement; or any other appropriate type of sensor. In the case of a machine such as a robot, observations from sensors may include observations of the position, linear or angular velocity, force, torque, or acceleration, or posture of one or more components of the machine, such as data characterizing the current state of the machine or robot or items held or handled by the machine or robot. Observations may also include, for example, sensed electronic signals, such as motor current or temperature signals, or image or video data, such as data from a camera or lidar sensor. Sensors such as these may be part of or located separately from the agents in the environment.

[0196] In some embodiments, the environment is a real-world environment of a service facility, including multiple electronic equipment items, such as a server farm or data center, such as a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include auxiliary control equipment that controls the operating environment of the equipment items, such as environmental control equipment, such as temperature control - for example, cooling equipment - or air flow control or air conditioning equipment - such as heaters, coolers, humidifiers - or other hardware that changes the properties of air in the real-world environment. Tasks may include tasks that control - for example, minimize - resource usage, such as tasks that control power consumption or water consumption. Agents may include electronic agents configured to control the operation of equipment items or control the operation of auxiliary - for example, environmental - control equipment.

[0197] In general, an action may be any action that has an effect on the observed state of the environment, for example, an action configured to adjust any of the sensed parameters described below. These actions may include actions that control an item of equipment or auxiliary control equipment, or that impose operating conditions on an item of equipment or auxiliary control equipment, for example, an action that causes a setting to be changed to adjust, control, or turn on or off the operation of an item of equipment or an auxiliary control equipment item.

[0198] Typically, observations of the state of an environment may include any electronic signal that represents the operation of a facility or the operation of equipment in the facility. For example, representations of the state of an environment may be derived from observations made by any sensor that senses the state of the physical environment of the facility or from observations made by any sensor that senses the state of one or more equipment items or one or more auxiliary control equipment items. These include sensors configured to sense electrical conditions such as current, voltage, power, or energy; the temperature of the facility; fluid flow, temperature, or pressure within the facility or within the cooling system of the facility; or physical facility configurations, such as whether a vent is open.

[0199] The reward or reward may be related to a measure of the performance of the task. For example, in the case of a task to control - e.g. minimise - the use of a resource - such as a task to control the use of electricity or water - the measure may include any measure of the use of the resource.

[0200] In some embodiments, the environment is a real-world environment of a power generation facility, for example, a renewable power generation facility, such as a solar farm or a wind farm. Tasks may include control tasks for controlling the power generated by the facility, for example, controlling the delivery of power to a distribution network, for example, to meet demand or reduce the risk of mismatches between grid elements, or to maximize the power generated by the facility. The agent may include an electronic agent configured to control the power generation of the facility or to couple the generated power into the grid. The action may include an action to control the electrical or mechanical configuration of the power generator, such as controlling the electrical or mechanical configuration of one or more renewable power generation elements, for example, controlling the configuration of a wind turbine or one or more solar panels or mirrors, or the electrical or mechanical configuration of a rotating generator. Mechanical control actions may, for example, include actions to control the conversion of energy input to electrical energy output, for example, the efficiency of the conversion or the degree of coupling of energy input to electrical energy output. Electrical control actions, for example, may include actions to control one or more of the voltage, current, frequency, or phase of the generated power.

[0201] The reward or reward may be related to a measure of the performance of the task. For example, in the case of a task to control the delivery of power to a distribution network, the metric may be related to a measure of the power delivered, or to a measure of an electrical mismatch between the generating facility and the grid, such as a voltage, current, frequency, or phase mismatch, or to a measure of power or energy lost in the generating facility. In the case of a task to maximize the delivery of power to a distribution network, the metric may be related to a measure of power or energy delivered to the grid, or to a measure of power or energy lost in the generating facility.

[0202] Typically, observations of the state of the environment may include any electronic signal representing the electrical or mechanical operation of the power generation equipment in the power generation facility. For example, representations of the state of the environment may be derived from observations of any sensor that senses the physical or electrical state of the equipment in the power generation facility that is generating electricity, or the physical environment of such equipment, or the condition of auxiliary equipment that supports the power generation equipment. Such sensors may include sensors configured to sense electrical conditions of the equipment such as current, voltage, power, or energy, temperature or cooling of the physical environment, fluid flow, or the physical configuration of the equipment; and observations of electrical conditions of the power grid, for example from local or remote sensors. Observations of the state of the environment may also include one or more predictions about future operating conditions of the power generation equipment, such as predictions of future wind or solar irradiance or predictions of future electrical conditions of the power grid.

[0203] As another example, the environment can be a chemical synthesis or protein folding environment, so that each state is a corresponding state of a protein chain or one or more intermediates or precursor chemicals, and the agent is a computer system for determining how to fold a protein chain or synthesize a chemical. In this example, the action is a possible folding action for folding a protein chain or an action for assembling a precursor chemical / intermediate, and the results to be achieved can include, for example, folding a protein to stabilize the protein and enable it to achieve a specific biological function or provide an effective synthetic pathway for a chemical. As another example, the agent can be a mechanical agent that indirectly performs or controls a protein folding action, such as by controlling a chemical synthesis step automatically selected by the system without human interaction. Observations can include direct or indirect observations of the state of a protein or chemical / intermediate / precursor and / or observations can be derived from simulations. Therefore, the system can be used to automatically synthesize proteins with specific functions, such as having a binding site shape, for example, binding with sufficient affinity to produce a biological effect so that it can be used as a ligand for a drug. For example, it can be an agonist or antagonist of a receptor or enzyme; or it can be an antibody that is configured to bind to an antibody target, such as a viral coat protein, or a protein expressed on a cancer cell, for example, to act as an agonist of a specific receptor or to prevent the binding of another ligand, thereby preventing activation of a related biological pathway.

[0204] Similarly, the environment can be a drug design environment, such that each state is a corresponding state of a potential pharmaceutically active compound, and the agent is a computer system for determining elements of the pharmaceutically active compound and / or a synthetic pathway for the pharmaceutically active compound. The drug / synthesis can be designed based on rewards derived from a drug target, such as in a simulation. As another example, the agent can be a mechanical agent that performs or controls the synthesis of the drug.

[0205] In some further applications, the environment is a real-world environment, and the agent manages the distribution of tasks across computing resources, such as on mobile devices and / or in data centers. In these embodiments, the operation may include assigning tasks to specific computing resources.

[0206] As further examples, an action may include presenting an advertisement, an observation may include an ad impression or a click count or click-through rate, and a reward may characterize a previous selection of an item or content by one or more users.

[0207] In some cases, observations may include textual or verbal instructions provided to the agent by a third party (e.g., an operator of the agent). For example, the agent may be an autonomous vehicle, and the user of the autonomous vehicle may provide textual or verbal instructions to the agent (e.g., to navigate to a particular location).

[0208] As another example, the environment may be an electrical, mechanical, or electromechanical design environment, e.g., an environment that simulates the design of an electrical, mechanical, or electromechanical entity. The simulation environment may be a simulation of a real-world environment in which the entity is intended to work. The task may be to design an entity. The observations may include observations that characterize the entity, i.e., observations of the mechanical shape or electrical, mechanical, or electromechanical configuration of the entity, or observations of parameters or properties of the entity. The actions may include actions that modify the entity, e.g., actions that modify one or more observations. The reward or reward may include one or more measures of the performance of the entity design. For example, the reward or reward may relate to one or more physical properties of the entity, such as weight or strength, or to one or more electrical properties of the entity, such as a measure of the efficiency of performing a specific function for which the entity is designed. The design process may include outputting a design for manufacture, e.g., in the form of computer executable instructions for manufacturing the entity. The process may include manufacturing the entity according to the design. Thus, the design of the entity may be optimized, e.g., by reinforcement learning, and then the optimized design for manufacturing the entity may be output, e.g., as computer executable instructions; the entity with the optimized design may then be manufactured.

[0209] As previously mentioned, the environment can be a simulated environment. Typically, in the case of a simulated environment, observations can include simulated versions of one or more previously described observations or observation types, and actions can include simulated versions of one or more previously described actions or action types. For example, the simulated environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent can be a simulated vehicle that navigates through the motion simulation. In these embodiments, the action can be a control input to control a simulated user or a simulated vehicle. Typically, the agent can be implemented as one or more computers that interact with the simulated environment.

[0210] The simulation environment can be a simulation of a specific real-world environment and agent. For example, the system can be used to select actions in the simulation environment during training or evaluation of the system, and after training or evaluation or both are completed, it can be deployed to control a real-world agent in a specific real-world environment that is the subject of the simulation. This can avoid unnecessary wear and damage to the real-world environment or the real-world agent, and can allow control neural networks to be trained and evaluated in situations that rarely occur or are difficult or unsafe to reproduce in the real-world environment. For example, the system can be partially trained using a simulation of a mechanical agent in a simulation of a specific real-world environment, and then deployed to control a real mechanical agent in a specific real-world environment. Therefore, in such a case, the observations of the simulation environment are related to the real-world environment, and the selected actions in the simulation environment are related to the actions to be performed by the mechanical agent in the real-world environment.

[0211] Optionally, in any of the above embodiments, observations at any given time step may include data from previous time steps that may be useful in characterizing the environment, such as actions performed at the previous time step, rewards received at the previous time step, or both.

[0212] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0213] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware - including the structures disclosed in this specification and their structural equivalents - or a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more computer program instruction modules encoded on a tangible non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagation signal, for example, a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0214] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or a plurality of processors or computers. The apparatus can also be or further include a dedicated logic circuit system, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus can optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0215] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, a single file dedicated to a related program, or multiple coordinated files, such as files storing one or more modules, subroutines or code portions. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0216] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components that are installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0217] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system such as an FPGA or ASIC, or by a combination of a dedicated logic circuit system and one or more programmed computers.

[0218] The computer suitable for executing computer programs can be based on a general or special microprocessor or both, or any other kind of central processing unit. Usually, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic element of a computer is a central processing unit for implementing or executing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory can be supplemented or incorporated therein by a special logic circuit system. Usually, the computer will also include or be operably coupled to receive data from one or more large-capacity storage devices for storing data such as a disk, a magneto-optical disk or an optical disk or transmit data thereto, or both. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0219] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0220] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, a keyboard and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. In addition, the computer can interact with the user by sending a text message or other form of message to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0221] The data processing apparatus for implementing the machine learning model can also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, namely inference workloads.

[0222] Ability to implement and deploy machine learning models using a machine learning framework such as the TensorFlow framework, Microsoft Cognitive Toolkit framework, Apache Singa framework, or Apache MXNet framework.

[0223] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, for example, as a data server, or includes a middleware component, for example, an application server, or includes a front-end component, for example, a client computer with a graphical user interface web browser or application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), for example, the Internet.

[0224] The computing system can include a client and a server. The client and the server are usually remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by means of computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data such as HTML pages to the user device, for example, to display data to a user interacting with the device acting as a client and to receive user input from the user. Data generated at the user device, such as the result of the user interaction, can be received from the device at the server.

[0225] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that can be claimed, but should be interpreted as descriptions of features specific to specific embodiments of specific inventions. Certain features described in the context of separate embodiments in this specification can also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features can be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features in the claimed combination can be deleted from the combination, and the claimed combination can be directed to a sub-combination or a variant of a sub-combination.

[0226] Similarly, although the operations are depicted in the figures and described in the claims in a particular order, it should not be understood as requiring that these operations be performed in the particular order shown or in a sequential order, or requiring that all of the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0227] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the operations recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

[0228] Various aspects of the present disclosure may be set forth in the following terms:

[0229] Clause 1. A method performed by one or more computers, the method comprising:

[0230] receiving a task description, wherein the task description is a natural language description of a task to be performed by an agent in an environment; and

[0231] Controlling the agent across a sequence of time steps so that the agent performs a task includes: at each of a plurality of time steps in the sequence:

[0232] obtaining a first observation representing a state of the environment at time step;

[0233] processing a reporter input comprising a first observation using a reporter neural network to generate a reporter output defining a natural language report characterizing progress of the agent in completing a task as of a time step;

[0234] processing a planner input including a task description and a natural language report using a planner neural network to generate natural language instructions for the agent;

[0235] obtaining a second observation characterizing a state of the environment at the time step;

[0236] processing the natural language instruction and the second observation using a policy neural network to generate a policy output defining an action to be performed by the agent;

[0237] Use the policy output to select actions; and

[0238] Make the agent perform the selected action.

[0239] Clause 2. A method according to any preceding clause, wherein the planner neural network has been pre-trained via unsupervised learning on a language modeling objective.

[0240] Clause 3. The method of clause 2, wherein the planner input further comprises one or more prompt inputs, each prompt input comprising:

[0241] Example task description;

[0242] one or more example reporter inputs; and

[0243] For each reporter input, an example natural language report generated in response to the reporter input.

[0244] Clause 4. A method according to any one of clauses 2 to 3, wherein the reporter output is a natural language report.

[0245] Clause 5. The method of any one of clauses 2 to 4, wherein the reporter output comprises a classification output across a plurality of categories.

[0246] Clause 6. The method according to clause 5, further comprising:

[0247] Use categorical output to select a category from multiple categories; and

[0248] A natural language report is generated, including a natural language description of the selected category inserted at a pre-position in the natural language report.

[0249] Clause 7. A method according to any preceding clause, further comprising:

[0250] receiving a task reward in response to the agent performing the selected action; and

[0251] The reporter neural network is trained via reinforcement learning using the received rewards.

[0252] Clause 8. The method of clause 7, wherein training the reporter neural network by reinforcement learning using the received reward comprises:

[0253] Train a reporter neural network without training a policy neural network.

[0254] Clause 9. The method of clause 8, wherein the policy neural network has been trained on a task different from the task described by the task description.

[0255] Clause 10. A method according to any of clauses 7 to 9, wherein the reporter neural network comprises one or more encoder neural networks that have been pre-trained.

[0256] Clause 11. A method according to clause 10, wherein the first observation comprises an image of the environment, wherein the reporter neural network comprises a first encoder neural network that is configured to process the image and has been pre-trained on a visual representation learning task.

[0257] Clause 12. A method according to clause 10 or clause 11, wherein the reporter input further includes natural language instructions from a previous time step, and wherein the reporter neural network includes a second encoder neural network that is configured to process natural language instructions and has been pre-trained on a text representation learning task.

[0258] Clause 13. The method of any of clauses 10 to 12, wherein the reporter neural network comprises an output subnetwork configured to receive a respective output from each of the encoder neural networks and to generate a reporter output from the respective output, and wherein training the reporter neural network comprises:

[0259] Train the output subnetwork via reinforcement learning while keeping the pre-trained encoder fixed; or

[0260] The output subnetwork and the pre-trained encoder are trained via reinforcement learning.

[0261] Clause 14. A method according to any preceding clause, wherein the first observation is the same as the second observation.

[0262] Clause 15. A method according to any preceding clause, wherein the agent is a mechanical agent and the environment is a real-world environment.

[0263] Clause 16. The method of clause 15, wherein the mechanical agent is a robot.

[0264] Clause 17. The method of clause 15 or clause 16, wherein the first observation and the second observation comprise data generated from sensor readings captured by one or more sensors of the mechanical agent.

[0265] Clause 18. A method according to any preceding clause, wherein obtaining a task description comprises:

[0266] A task description is obtained as text input or verbal input from the user.

[0267] Clause 19. A method according to any preceding clause, wherein the planner input further comprises one or more natural language reports from one or more previous time steps in the sequence.

[0268] Clause 20. A method according to any preceding clause, further comprising:

[0269] Obtaining demonstration data, the demonstration data comprising a natural language description of an example task and data representing performance of the example task by an expert agent; and

[0270] The reporter neural network is trained by imitation learning on the demonstration data.

[0271] Clause 21. The method of clause 20, wherein training the reporter neural network by imitation learning on the demonstration data comprises:

[0272] Train the reporter neural network on the demonstration data while keeping the policy neural network and planner neural network fixed.

[0273] Clause 22. The method of any preceding clause, wherein the environment is a real-world environment of a service facility including a plurality of electronic equipment items, and the agent is an electronic agent configured to control operation of the service facility.

[0274] Clause 23. A method according to any preceding clause, wherein the environment is a real-world manufacturing environment for manufacturing a product, and the agent comprises an electronic agent configured to control operation of a manufacturing unit or machine for manufacturing the product.

[0275] Clause 24. A system comprising:

[0276] one or more computers; and

[0277] One or more storage devices communicatively coupled to one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of clauses 1 to 23.

[0278] Clause 25. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of clauses 1 to 23.< / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos>

Claims

1. A method performed by one or more computers, the method comprising: receiving a task description, wherein the task description is a natural language description of a task to be performed by an agent in an environment; and Controlling the agent across a sequence of time steps so that the agent performs the task comprises: at each time step in a plurality of time steps in the sequence: obtaining a first observation characterizing a state of the environment at the time step; processing, using a reporter neural network, a reporter input comprising the first observation to generate a reporter output defining a natural language report characterizing progress of the agent in completing the task as of the time step; processing a planner input including the task description and the natural language report using a planner neural network to generate natural language instructions for the agent; obtaining a second observation characterizing a state of the environment at the time step; processing the natural language instruction and the second observation using a policy neural network to generate a policy output defining an action to be performed by the agent; Selecting an action using the policy output; and Causes the agent to perform the selected action.

2. A method according to any preceding claim, wherein: The planner neural network has been pre-trained via unsupervised learning on a language modeling objective.

3. The method according to claim 2, wherein: The planner input further includes one or more prompt inputs, each prompt input including: Example task description; one or more example reporter inputs; and For each reporter input, an example natural language report generated in response to that reporter input.

4. The method according to any one of claims 2 to 3, wherein: The reporter output is the natural language report.

5. The method according to any one of claims 2 to 4, wherein: The reporter output includes classification output across multiple categories.

6. The method according to claim 5, further comprising: selecting a category from the plurality of categories using the classification output; as well as Generating the natural language report includes inserting the natural language description of the selected category at a predetermined location in the natural language report.

7. The method according to any preceding claim, further comprising: receiving a reward for the task in response to the agent performing the selected action; as well as The reporter neural network is trained by reinforcement learning using the received rewards.

8. The method according to claim 7, wherein: Training the reporter neural network by reinforcement learning using the received rewards includes: The reporter neural network is trained without training the policy neural network.

9. The method according to claim 8, wherein: The policy neural network has been trained on a task different from the task described by the task description.

10. The method according to any one of claims 7 to 9, wherein: The reporter neural network includes one or more encoder neural networks that have been pre-trained.

11. The method according to claim 10, wherein: The first observation comprises an image of the environment, wherein the reporter neural network comprises a first encoder neural network that is configured to process the image and that has been pre-trained on a visual representation learning task.

12. The method according to claim 10 or claim 11, wherein: The reporter input further comprises a natural language instruction from a previous time step, and wherein the reporter neural network comprises a second encoder neural network configured to process the natural language instruction and having been pre-trained on a text representation learning task.

13. The method according to any one of claims 10 to 12, wherein: The reporter neural network comprises an output subnetwork configured to receive a respective output from each of the encoder neural networks and to generate a reporter output from the respective outputs, and wherein training the reporter neural network comprises: Training the output subnetwork via reinforcement learning while keeping the pre-trained encoder fixed; or The output sub-network and the pre-trained encoder are trained by reinforcement learning.

14. A method according to any preceding claim, wherein: The first observation is the same as the second observation.

15. A method according to any preceding claim, wherein: The agent is a mechanical agent and the environment is a real-world environment.

16. The method according to claim 15, wherein: The mechanical agent is a robot.

17. The method according to claim 15 or claim 16, wherein: The first observation and the second observation include data generated from sensor readings captured by one or more sensors of the mechanical agent.

18. A method according to any preceding claim, wherein: Obtaining the task description includes: The task description is obtained as text input or verbal input from a user.

19. A method according to any preceding claim, wherein: The planner input further includes one or more natural language reports from one or more previous time steps in the sequence.

20. The method according to any preceding claim, further comprising: Obtaining demonstration data, the demonstration data comprising a natural language description of an example task and data representing performance of the example task by an expert agent; as well as The reporter neural network is trained by imitation learning on the demonstration data.

21. The method according to claim 20, wherein: Training the reporter neural network by imitation learning on the demonstration data includes: The reporter neural network is trained on the demonstration data while keeping the policy neural network and the planner neural network fixed.

22. A method according to any preceding claim, wherein: The environment is a real-world environment of a service facility including a plurality of electronic equipment items, and the agent is an electronic agent configured to control the operation of the service facility.

23. A method according to any preceding claim, wherein: The environment is a real-world manufacturing environment for manufacturing a product, and the agents include electronic agents configured to control operations of a manufacturing unit or machine for manufacturing the product.

24. A system comprising: one or more computers; as well as One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method according to any one of claims 1 to 23.

25. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 23.

Images