Training advanced controllers to generate natural language commands for controlling agents

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training advanced controller neural networks for controlling agents. Specifically, a high-level controller neural network generates a natural language command, which may be provided as input to a low-level controller neural network that generates a control output that may be used to control an agent.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 584,156, filed September 20, 2023. The disclosure of the earlier application is considered part of the disclosure of this application and is incorporated herein by reference. Background Technology

[0003] This specification relates to the use of neural networks to control intelligent agents.

[0004] Machine learning models receive input and generate outputs based on the received inputs, such as predicted outputs. Some machine learning models are parametric models, and generate outputs based on the received inputs and the values ​​of the model parameters.

[0005] Some machine learning models are deep models, which employ multiple layers to generate outputs from received inputs. For example, a deep neural network is such a deep machine learning model: it includes an output layer and one or more hidden layers, each of which applies a non-linear transformation to the received input to generate an output. Summary of the Invention

[0006] This specification generally describes a system implemented as a computer program on one or more computers at one or more locations, which controls an intelligent agent interacting in an environment based on outputs generated by a high-level controller neural network and a low-level controller neural network. Specifically, the high-level controller neural network generates natural language commands, which are then processed by the low-level controller neural network to generate control inputs for the intelligent agent.

[0007] Generally, this disclosure proposes that an agent interacting with an environment (e.g., a real-world environment) is controlled by: a first "high-level" controller neural network trained to receive input including observations characterizing the state of the environment at a specific time (time step, which is one time step in a sequence of time steps), and generating outputs defining natural language commands based on the inputs; and a second "low-level" controller neural network processing the natural language commands to generate control outputs for controlling the agent. The high-level controller can be trained by a method comprising: obtaining a training dataset including a plurality of demonstration trajectories, each demonstration trajectory including, for each of the plurality of time steps, a corresponding observation characterizing the state of a demonstration environment with which the demonstration agent is interacting at that time step and a corresponding natural language command provided to the demonstration agent at that time step; and training the high-level controller neural network on the demonstration trajectories in the training dataset through supervised learning.

[0008] Generating this type of demonstration trajectory is relatively inexpensive, such as by using natural language commands to generate the demonstration trajectory manually. These natural language commands can break down the task performed in the corresponding demonstration trajectory into subtasks associated with the corresponding natural language commands within that natural language command. This allows for the benefit of human understanding of the task. A possible consequence is that a control system employing a trained high-level controller to control a low-level controller (such as a separately trained low-level controller) can be able to control the agent to perform complex tasks (e.g., tasks different from those performed in the corresponding demonstration trajectory) with a high success rate, such as tasks optimally performed by executing long sequences of subtasks. In other words, the high-level controller can be able to simulate the human ability to break down the task expected of the agent into simpler subtasks, and the low-level controller can be trained to perform simpler subtasks, which can be easier and more successful than training it to perform complex tasks.

[0009] Specific embodiments of the subject matter described in this specification may be implemented in order to achieve one or more of the following advantages.

[0010] Despite recent successes in reinforcement learning (RL), a major challenge remains using neural networks trained via RL to control agents in real-world settings. For example, traditional RL has historically been challenging in learning goal-oriented behavior over long timeframes, particularly given the relatively data-hungry exploration process and time contribution assignments required by RL. This is especially limiting in real-world or embody-like tasks where motor control operations in an action space require long sequences of motor actions, even for relatively simple tasks.

[0011] In particular, Reinforcement (RL) has primarily flourished in worlds that provide simple, abstract action spaces, such as games, where a single "action" causes significant changes in the environment. However, this is limiting. For example, the main advantage of generalized embodied action spaces is that they are realistic, flexible, and permit open-ended and emergent behavior. The inability of RL to operate in these action spaces (due to the challenges of exploration and long-term contribution assignment on long action sequences) has become a major obstacle to the application of RL in real-world settings.

[0012] Hierarchical reinforcement learning (RL)—a technique that uses low-level and high-level controllers to control agents—has emerged as a compelling approach to achieving goal-oriented behavior over long sequences of actions and to address the aforementioned problems using “flat” reinforcement learning methods. Intuitively, this means that the “action space” on which we actually assign and explore contributions is a sequence of actions that extends in time toward achieving sub-goals on the path to the goal task. The main challenge here has been designing (or learning) a sufficiently broad sub-goal space that is both efficient in reducing the scope of planning and sufficiently expressive to permit the behavior of interest. Specifically, the core challenge is finding an appropriate set of abstractions for a given domain and set of tasks. For example, many existing methods represent sub-goals generated as output by the high-level controller (to be provided as input to the low-level controller) as latent vectors of numerical values. However, the resulting vectors are uninterpretable, and learning sub-goals that efficiently reduce the scope of planning becomes difficult.

[0013] Alternatively, this specification describes the use of natural language as a means of parameterizing this sub-goal space. Language is a lossy channel—a textual description of an agent's trajectory will discard much (detailed, basic, visual) information. However, language has explicitly evolved to still be sufficiently expressive to represent the vast majority of thoughts, goals, and behaviors relevant to humans. This makes language a strong contender for specifying sub-goals that both effectively reduce complexity and retain expressiveness where important. Language also has the added advantage of readily available training data from, for example, human users or other agents. Specifically, this specification describes how to train a high-level controller to efficiently generate natural language commands that can be processed by a low-level controller to control the agent. Specifically, this specification describes how the system can use data including natural language commands for the agent to provide soft supervision to a hierarchical agent that can learn, for example, to solve complex long-term tasks in a 3D embodied environment. That is, by utilizing the described training techniques, the system can train a high-level controller to communicate effectively with a low-level controller via natural language.

[0014] Additionally, using unconstrained natural language to parameterize the sub-objective space has additional advantages. For example, generating demonstration data, such as from human participants, is straightforward. As another example, it provides sufficient flexibility to represent a large number of sub-objectives in human-related tasks. As yet another example, natural language commands can be interpreted by users (e.g., those monitoring the performance of the agent).

[0015] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0016] Figure 1 An example action selection system is shown.

[0017] Figure 2 This is a flowchart of an example process for controlling an agent at a given time step.

[0018] Figure 3 This is a flowchart of an example process for training an advanced controller neural network.

[0019] Figure 4 An example of system operation is shown.

[0020] Figure 5 An example of the performance of the described technology is shown.

[0021] In the various figures, the same reference numerals and names indicate the same elements. Detailed Implementation

[0022] Figure 1 An example action selection system 100 is shown. Action selection system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below are implemented.

[0023] The action selection system 100 uses a high-level controller neural network 120 and a low-level controller neural network 130 to control an agent 104 interacting with an environment 106 to perform tasks within the environment 106. For example, agent 104 can be a robot, such as a robotic arm, quadruped robot, humanoid robot, or other types of robots that can be controlled by system 100. Agent 104 can also be different types of agents, such as control systems for facilities, software agents, etc. As a specific example, agent 104 can be an avatar or other character in a video game environment. For example, one or more other agents within a video game environment can be controlled by a user, while agent 104 is controlled by system 100.

[0024] Examples of agents, environments, and tasks will be described below.

[0025] When controlling agent 104, system 100 controls agent 104 to complete the task by selecting action 108 to be performed by agent 104 at each of multiple time steps during the execution of a task episode.

[0026] A task “round” is a sequence of interactions during which an agent attempts to perform a task starting from a certain initial state of the environment. In other words, each task round begins with the environment in an initial state, such as a fixed initial state or a randomly chosen initial state, and ends when the agent has successfully completed the task or when a termination criterion is met, such as the environment entering a state that has been designated as the termination state or the agent performing a threshold number of actions without successfully completing the task.

[0027] At each time step during any given task round, system 100 receives observations 110 characterizing the current state of environment 106 at that time step, and in response, selects an action 108 to be performed by agent 104 at that time step. After agent 104 performs action 108, environment 106 transitions to the new state.

[0028] Observation 110 may include any appropriate information characterizing the state of the environment. As an example, observation 110 may include sensor readings from one or more sensors configured to sense the environment. For example, observation 110 may include one or more images captured by one or more cameras, measurements from one or more proprioceptive sensors, etc.

[0029] In some cases, system 100 receives an extrinsic reward 150 (also known as a “task” reward) from the environment in response to the agent performing an action.

[0030] Generally speaking, the reward is a scalar value and represents the agent's progress toward completing the task.

[0031] As a specific example, the reward can be a sparse binary reward, which is zero unless the task is successfully completed, and one if the task is successfully completed due to the action being performed.

[0032] As another specific example, the reward can be a dense reward, which measures the agent's progress toward completing the task when individual observations are received during the rounds in which the task is attempted, i.e., such that a non-zero reward can and often is received before the task is successfully completed.

[0033] In some cases, rewards can be generated by reward models such as those based on observations. As an example of this, a success detector can be used to learn a reward model that detects successful behaviors from observations of the environment, for example, as described in "Vision-Language Models as Success Detectors" arXiv:2303.07280.

[0034] Generally, when controlling agent 104, system 100 uses high-level controller neural network 120 to generate natural language commands 122, while using low-level controller to generate control outputs 132 for controlling agent 104.

[0035] More specifically, at each time step, system 100 receives observations 110 that characterize the state of environment 106 at that time step.

[0036] System 100 uses an advanced controller neural network 120 to generate natural language commands 122 for time steps based on observations 110.

[0037] In some cases, system 100 may not need to generate a new command at every time step. In these cases, at some time steps, system 100 may use the most recently generated natural language command 122 as the command for that time step, while at other time steps, the system processes observation 110 to generate a new command for that time step. That is, system 100 may reuse the most recently received natural language command 122 until a new command 122 is generated by the higher-level controller 120.

[0038] The “natural language command” 122 is called “natural language” because it is a sequence of text in a natural language, such as English, French, or Spanish, specifying instructions that the agent can follow. The commands typically specify high-level interactions with the environment, rather than low-level input for controlling the agent. Examples of commands include “move forward,” “drop it,” “grasp the cube,” etc. In other words, as part of performing a task, the natural language command 122 can specify sub-goals to be achieved by the agent in natural language.

[0039] Therefore, the controller neural network 120 is referred to as a "high-level" controller because the controller generates high-level outputs that do not directly specify the control inputs for the agent 104, but instead specify "high-level" instructions for the agent. The controller neural network 120 can be equivalently referred to as a first controller neural network.

[0040] The high-level controller 120 is a neural network configured to receive inputs including observations of the state of the environment with which the agent is interacting; and to generate outputs that define natural language commands for a low-level controller neural network, which generates control outputs for controlling the agent.

[0041] The input to the advanced controller 120 may optionally include additional data besides observations. For example, system 100 may receive natural language instructions or other communications specifying a task to be performed, for instance, from a user or from the environment. In this example, the input to the advanced controller 120 may also include the most recent communications. As another example, the input may include one or more previous natural language commands that have already been generated at one or more previous time steps.

[0042] The advanced controller 120 can typically have any suitable architecture that allows the neural network to map inputs, including observations, to natural language commands.

[0043] For example, the advanced controller 120 may include an encoder neural network that encodes an input to generate an encoded representation of the input; and then includes an additional layer that operates on the encoded representation to generate a command.

[0044] For example, the additional layer may include one or more self-attention layers that operate on the encoded representation. Examples of suitable architectures for self-attention layers are described in Ashish Vaswani et al., “Attention is all you need,” Advances in Neural Information Processing Systems, pp. 5998–6008, 2017; arXiv:1810.04805 Devlin et al. (BERT); and arXiv:1901.02860 Dai et al. (Transformer-XL). For example, a self-attention layer may include an attention mechanism defined by a query (“Q”) matrix, a key (“key”) matrix, and a value (“V”) matrix, composed of trained numerical parameters, and configured to generate a function-weighted output based on the product of the input sequence and the value matrix, and the product of the input sequence and the key matrix.

[0045] As another example, additional layers may include one or more recurrent neural network layers that operate on the encoded representation.

[0046] As a specific example, including a recurrent layer or a self-attention layer in the advanced controller 120 can allow the advanced controller 120 to be conditional on previous observations received at a previous time step.

[0047] As a specific example, when the observation is an image and the controller also receives natural language instructions, the high-level controller 120 may include: an image encoder neural network, such as a convolutional neural network or a Transformer neural network including one or more self-attention layers; a text encoder neural network, such as a recurrent neural network or a Transformer neural network including one or more self-attention layers; and a multimodal neural network that combines the outputs of the image and text encoders to generate a composite representation. For example, the multimodal neural network may be implemented as one or more self-attention layers that apply attention to the outputs of the image and text encoders. When the high-level controller 120 does not receive text, the high-level controller 120 may include only an image encoder, and the output of the image encoder may be considered a composite representation. The high-level controller 120 may then include recurrent neural network layers, such as long short-term memory (LSTM) layers incorporating context from previous observations, and then include a text decoder, such as a recurrent neural network or a Transformer neural network, to generate natural language commands.

[0048] System 100 then uses a low-level controller neural network 130 to process inputs including at least natural language commands 122 for time steps to generate control outputs 132 for controlling agent 104 at time steps.

[0049] The low-level controller neural network 130 is a neural network configured to receive input including natural language commands and process the input to generate control output 132 for controlling the agent 104.

[0050] Control output 132 specifies inputs for one or more controls of agent 104. In other words, control output 132 is a "low-level" output that can be directly used to control the agent. For example, when agent 104 is a robot, the low-level output may include corresponding inputs for each of a plurality of actuators, joints, or other controllable elements of the robot.

[0051] Therefore, the controller neural network 130 is referred to as a "low" level controller because it generates low-level outputs that directly specify the control inputs for the agent 104, rather than specifying "high-level" instructions for the agent. The controller neural network 130 can be equivalently referred to as a second controller neural network.

[0052] The input to the low-level controller neural network 130 may optionally include other data. For example, the input to the low-level controller 130 may include observation 110 at a time step. As another example, the input to the low-level controller 130 may include a portion of observation 110 at a time step. For example, when observation 110 includes data from multiple different sensors, the input to the low-level controller 130 may include a proper subset of data from the sensors, such as data from only the proprioceptive sensor and not from the sensor that generates higher-dimensional outputs—e.g., an image or lidar sensor.

[0053] As another example, the input to the low-level controller neural network 130 may include instructions or other communications received from the environment or from the user.

[0054] The low-level controller neural network 130 can typically have any architecture that allows the neural network 130 to map inputs to control outputs 132. For example, the low-level controller neural network 130 may include an encoder neural network for encoding the inputs and one or more additional layers for generating control outputs 312 based on the encoded inputs. For example, the low-level controller neural network 130 may share an encoder with the high-level controller 120.

[0055] As a specific example, the low-level controller neural network 130 may have the same architecture as the example architecture given above for the high-level controller 120, but in which the text decoder is replaced by a decoder that generates the control output—such as a multilayer perceptron (MLP), RNN, or Transformer neural network.

[0056] In order for the high-level controller 120 to generate commands 122 that the low-level controller 130 can use to accurately execute tasks in environment 106, system 100 trains the high-level controller 120.

[0057] In some implementations, system 100 trains a high-level controller 120 through supervised learning on a set of demonstration trajectories. For each of the plurality of time steps, each demonstration trajectory then includes a corresponding observation characterizing the state of the demonstration environment with which the demonstration agent (e.g., an agent of the same form as agent 104) is interacting at that time step, and a corresponding natural language command provided to the demonstration agent at that time step.

[0058] In some other implementations, system 100 trains high-level controller 120 using reinforcement learning, for example by iteratively modifying a function of the high-level controller to maximize reward 150, such as the expected value of the reward function at multiple time steps in an epoch. When training with reinforcement learning, system 100 uses high-level controller 120 to control the agent and uses the resulting rewards to train high-level controller 120.

[0059] In other implementations, system 100 trains the advanced controller 120 through both supervised learning and reinforcement learning.

[0060] When training using both supervised learning and reinforcement learning, system 100 can train the high-level controller 120 first through supervised learning and then through reinforcement learning. Alternatively, system 100 can jointly train the high-level controller 120 using both supervised learning and reinforcement learning, for example, with respect to a total loss function that is the sum or weighted sum of the supervised learning loss and the reinforcement loss. That is, each training step may include updating the controller's parameters using the gradients of both losses.

[0061] Generally, whether training is performed using supervised learning or reinforcement learning, system 100 or another training system first pre-trains the low-level controller 130, and then system 100 keeps the low-level controller 130 fixed during the training of the high-level controller 120. For example, system 100 or other training systems can train the low-level controller 130 through supervised learning, such as through imitation learning. See below. Figure 3 An example illustrating this situation. As another example, system 100 or other training system 100 may utilize different high-level controllers to train a low-level controller 120 by using a low-level controller 120 that controls an agent conditioned on commands generated by different high-level controllers (e.g., a neural network that has been trained or a fixed policy that maps time steps in a trajectory to commands).

[0062] The following is for reference. Figure 3 and Figure 4 The training of the high-level controller 120 and the low-level controller 130 is described in more detail.

[0063] Figure 2 This is a flowchart of an example process 200 for controlling an agent at a given time step during a task round. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to this specification, such as... Figure 1 The action selection system 100 can execute process 200.

[0064] The system receives observations characterizing the state of the environment at that time step (step 202).

[0065] The system uses an advanced controller neural network to generate natural language commands for that time step (step 204).

[0066] As mentioned above, a "natural language command" is called "natural language" because it is a sequence of text in a natural language, such as English, French, or Spanish, that specifies commands that can be followed by an agent. In other words, as part of performing a task, a natural language command can specify sub-goals to be achieved by the agent in natural language.

[0067] In some cases, the system may not need to generate a new command at every time step. In these cases, at some time steps, the system can use the most recently generated natural language command as the command for that time step. That is, the system can reuse the most recently received natural language command until a new command is generated.

[0068] Specifically, when one or more criteria for generating new commands are met, the system can use a high-level controller neural network to process inputs including observations at time steps to generate natural language commands for those time steps.

[0069] The system can determine whether a criterion is met in any of a variety of ways. As an example, the system can determine each... k Several time steps meet this standard, among which k It is an integer greater than 1. As another example, the system can use the learned model or heuristics to determine, based on observation or other data available from the environment, whether a previous natural language command has been successfully executed by the agent, and if so, whether the criterion has been met.

[0070] The system then uses a low-level controller neural network to process at least the natural language commands to generate control outputs for controlling the agent at that time step (step 206).

[0071] As described above, control outputs are "low-level" outputs that can be directly used to control an agent. For example, when the agent is a robot, low-level outputs may include corresponding inputs for each of the robot's multiple actuators, joints, or other controllable elements.

[0072] Figure 3 This is a flowchart of an example process 300 for training an advanced controller neural network. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to this specification, such as... Figure 1 The action selection system 100 can execute process 300.

[0073] The system obtains data from the specified pre-trained low-level controller neural network (step 302).

[0074] In other words, the system or another training system pre-trains the low-level controller neural network before training the high-level controller neural network.

[0075] Specifically, the low-level controller neural network can be pre-trained through supervised learning on multiple low-level demonstration trajectories.

[0076] For each of the multiple time steps, each low-level demonstration trajectory includes (i) a corresponding observation characterizing the state of the demonstration environment with which the demonstration agent is interacting at that time step, (ii) a corresponding natural language command provided to the demonstration agent at that time step, and (iii) an action performed by the demonstration agent at that time step. The action performed by the agent is a low-level action in the same space as the control output generated by the low-level controller.

[0077] For example, the system can obtain these low-level trajectories by recording the interactions between the demonstration agent and the demonstration environment.

[0078] For example, the demonstration agent can be, for instance, an agent controlled by a fixed, learned policy, an agent controlled by a user, or a human user performing a task in an environment. Specifically, at each time step, the demonstration agent or the user controlling the demonstration agent can be provided with a corresponding natural language command, and the demonstration agent can then perform an action at that time step.

[0079] As a specific example, the system can collect low-level trajectories based on interactions with a demonstration environment of both a "setter" agent and a "solver" agent to perform a set of tasks. For a given task, a single controllable agent is controlled by the "solver." Given a task objective, the "setter" instructs the "solver" on how to solve the task, such as through a chat interface. The "setter" can observe the "solver" but cannot directly interact with the environment. For example, both the "setter" and the "solver," and the solver itself, can be controlled by the user, or the "setter" can be controlled by the user while the "solver" is controlled by a natural language conditional policy that receives natural language input and outputs control policies.

[0080] As an example, a low-level controller can be trained on a low-level demonstration trajectory via behavior cloning. When trained via behavior cloning, the system can train the low-level controller to minimize the behavior cloning loss, which, for each of multiple time steps in each low-level demonstration trajectory, measures the probability that the corresponding demonstration action, assigned to the demonstration agent at that time step, is produced by the output generated by the low-level controller's neural network through processing inputs including the corresponding command for that time step. For example, when the low-level controller also receives observation o as input, the supervised training objective could be: , Where π represents the low-level controller, g is the natural language command, and a is the demonstration action. B It is the number of tracks in the batch (indexed by integer). n (mark), and K It is the number of time steps in the trajectory (indexed by integer). t mark).

[0081] Then, the system trains the high-level controller neural network.

[0082] As an example, the system can train an advanced controller neural network through supervised learning (step 304).

[0083] Training through supervised learning typically involves obtaining a training dataset that includes multiple demonstration trajectories.

[0084] For each of the multiple time steps, each demonstration trajectory then includes a corresponding observation characterizing the state of the demonstration environment with which the demonstration agent is interacting at that time step, and a corresponding natural language command provided to the demonstration agent at that time step.

[0085] For example, a demo trajectory can be generated based on the same interactions as a low-level demo trajectory, or based on demo data from different groups.

[0086] Then, the system trains a high-level controller neural network on demonstration trajectories in the training dataset through supervised learning, such as behavioral cloning.

[0087] For example, when using behavioral cloning, the system can be trained on a loss that, for each of multiple time steps in each demonstration trajectory, measures the probability that the corresponding natural language command provided to the demonstration agent at that time step is assigned by the output generated by a high-level controller neural network through processing inputs including the corresponding observations for that time step. For example, the objective could satisfy:

[0088] in Represents a high-level controller neural network, and B, n, K, t g and o are defined as above.

[0089] As another example, the system can train an advanced controller neural network through reinforcement learning (step 304).

[0090] Training a high-level controller typically involves controlling the agent by executing process 200 at each of a plurality of first time steps in a reinforcement learning trajectory, thereby generating the reinforcement learning trajectory. A “first” time step is a time step in which the criteria for generating new natural language commands are met.

[0091] In other words, at each “first” time step where the criteria are met, the system can perform the following: receive a current observation representing the state of the training environment with which the training agent is interacting; use a high-level controller neural network to process the input, including the current observation, to generate an output that defines a natural language command for the first time step; use a low-level controller neural network to process the input, including the natural language command for the first time step, to generate a control output for the first time step; and use the control output to control the agent.

[0092] The system then receives the reward for the first time step. Therefore, for each first time step, the trajectory typically includes observations, natural language commands, and a reward.

[0093] Then, the system uses at least the reward from the first time step to train the high-level controller neural network through reinforcement learning. That is, although the reward is generated as a result of the control output produced by the low-level controller neural network, the system associates the reward with the corresponding natural language command generated by the high-level controller neural network.

[0094] In some cases, the system also trains a high-level controller neural network on the reward of a "second" time step, where the criteria are not met and the most recent natural language command is reused. That is, the trajectory includes both a "first" time step and a "second" time step. In these cases, at each of the second time steps in the reinforcement learning trajectory, the system receives a current observation representing the state of the training environment with which the trained agent is interacting; determines that the criteria for generating new natural language commands are not met; and, in response, uses a low-level controller neural network to process input including the natural language command from the most recent first time step to generate a control output for the second time step; and uses this control output to control the trained agent. The system then receives the reward for the second time step. Therefore, for each second time step, the trajectory typically includes an observation, the most recent natural language command, and a reward.

[0095] The system can typically train high-level control on any suitable reinforcement learning objective to maximize the expected reward. Examples of such objectives include policy gradient algorithms, actor-critic algorithms, etc.

[0096] As a specific example, when the system uses the V-Trace objective, it augments the neural network with value headers and optimizes: , Where R n,t It is in the environment where the trajectory n Time step in tThe reward received by the agent after observing and representing the state at a certain point, and V n,t It is the value head in the trajectory n Time step in t The value score generated at that point. The value score typically measures the performance of an environment at a given time step. t The estimated reward is received by the agent after the observed state is represented. The reward is measured on the trajectory. n The combination of rewards received at future time steps, such as a sum or a sum discounted over time.

[0097] Because the low-level controller is frozen during reinforcement learning training, the high-level and low-level controllers cannot develop different communication protocols via RL—the high-level controller is restricted to using commands that the low-level controller, trained on natural language instructions, can understand. This increases the interpretability of the agent's behavior after training.

[0098] As another example, the system can train the high-level controller neural network using both reinforcement learning and supervised learning (step 308).

[0099] When both supervised learning and reinforcement learning are used for training, the system can first train the high-level controller through supervised learning and then through reinforcement learning.

[0100] Alternatively, the system can jointly train a high-level controller using both supervised learning and reinforcement learning, for example, on a total loss function that is the sum or weighted sum of the supervised learning loss and the reinforcement loss. That is, each training step can include updating the controller's parameters using the gradients of both losses.

[0101] For example, the total loss function can satisfy:

[0102] Where w BC This refers to the weights assigned to the supervised learning objective, and w RL It refers to the weight assigned to the reinforcement learning objective.

[0103] Figure 4 Example 400 of system operation is shown.

[0104] In Example 400, both the high-level controller and the low-level controller receive input including observed images of the environment. In Example 400, the images are first-person images captured by the robot's camera.

[0105] Specifically, Figure 4Example 410 is illustrated for observations, outputs, and losses for a low-level controller 130. As shown in Example 410, the low-level controller receives inputs including images and text instructions and processes these inputs to generate a control output including a 10-dimensional action vector, for example, the 10-dimensional action vector including the corresponding control inputs for each of a set of actuators for a robot. During training of the low-level controller, this control output is used to compute a behavior cloning (BC) loss, which measures the difference between the control output and an output specifying the action to be performed by the demonstration agent given the observations and text instructions.

[0106] Figure 4 Example 420 of observations, outputs, and losses for the high-level controller 120 is also shown. As shown in example 420, the high-level controller receives input including an image and processes that input to generate a natural language text command. During the training of the high-level controller, this command is used to compute the Behavior Cloning (BC) loss and the Reinforcement Learning (RL) loss.

[0107] Figure 4 Example 430 is also shown, illustrating the observational input provided to the advanced controller 120 as input during a round of tasks requiring the robot to find and consume apples, and the corresponding natural language commands generated by the advanced controller 120 as output. As can be seen from the example, the command varies drastically throughout the round based on the context provided by the observed image.

[0108] Figure 5 An example of the performance of the described technology is shown.

[0109] Specifically, Figure 5 Example 510 illustrates the performance of the described technique (“hierarchical”) relative to a technique using a “flat” architecture to control an agent, which uses a single neural network to directly map observations to control outputs. As can be seen from Example 510, while the flat architecture can control the agent to perform simple tasks, the described technique significantly outperforms the flat architecture on difficult tasks. Therefore, the results demonstrate that the described hierarchical structure allows the system to successfully enable the agent to complete longer and more complex tasks than might be possible without a hierarchical structure.

[0110] Figure 5 Example 520 also illustrates the performance of various examples of the described techniques. More specifically, Example 520 shows the performance with various ratios of behavioral cloning (BC) versus reinforcement learning (RL) training. For example, these ratios can represent w BC With w RL Different ratios, the w RLThis refers to the weights assigned to the reinforcement learning objective. As can be seen from Example 520, training using both behavioral cloning and RL is generally superior to using only BC or RL, especially for difficult tasks.

[0111] As described above, the system can use various environments to train and control both high-level and low-level controller neural networks. For example, the system can use data collected from a demonstration environment to train the neural network via supervised learning. As another example, the system can control the agent in a training environment as part of training via reinforcement learning. As yet another example, the system can control the agent in an "inference" environment after training.

[0112] These environments can be the same or different.

[0113] For example, the demonstration environment can be the same as the inference environment, or it can be a different environment. As a specific example, the demonstration environment can be the simulated environment, and the inference environment can be the simulated environment, or it can be a real-world environment simulated by the demonstration environment.

[0114] As another example, the training environment can be the same as the inference environment, or it can be a different environment. As a specific example, the training environment can be a simulated environment, and the inference environment can be a simulated environment, or it can be a real-world environment simulated by the training environment.

[0115] Here are some examples of the types of intelligent agents that the system can currently control.

[0116] In some implementations, the environment is a real-world environment, and the agent is a mechanical intelligent agent that interacts with the real-world environment. Examples include robots that operate or navigate through the environment, or autonomous or semi-autonomous land, air, or sea vehicles. The actions are performed by the mechanical intelligent agent in the real-world environment to execute a task. For instance, the agent could be a robot that interacts with the environment to complete a specific task (e.g., locating an object of interest in the environment, moving the object of interest to a designated location in the environment, or navigating to a designated destination in the environment).

[0117] In these implementations, observation can include one or more of the following: images, object position data, and sensor data, to capture observations as the agent interacts with its environment, such as sensor data from images, distance or position sensors, or from actuators. For example, in the case of a robot, observation can include data characterizing the robot's current state, such as one or more of the following: joint positions, joint velocities, joint forces, torque or acceleration (e.g., gravity-compensated torque feedback), and the global or relative pose of an object held by the robot. In the case of a robot or other mechanical agent or vehicle, observation can similarly include one or more of the following: the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. Observation can be defined in one, two, or three dimensions and can be absolute and / or relative. Observation can also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or image or video data, such as data from sensors of the agent or from sensors located separately from the agent in its environment.

[0118] In these implementations, actions can be control signals for controlling a robot or other mechanical agent, such as torque or higher-level control commands for a robot's joints; or control signals for controlling autonomous or semi-autonomous land, air, or sea vehicles, such as torque or higher-level control commands for controlling surfaces or other control elements (e.g., steering control elements of a vehicle). Control signals can include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or parts of another mechanical agent (typically an electromechanical agent). Control signals can also, or alternatively, include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within an environment, the control of which affects the observed state of the environment. For example, in the case of autonomous or semi-autonomous land, air, or sea vehicles, control signals can define actions for controlling navigation (e.g., steering) and movement (e.g., braking and / or acceleration of the vehicle).

[0119] In some implementations, the environment is a simulation of the aforementioned real-world environment, and the agent is implemented as one or more computers interacting with the simulated environment. For example, the simulated environment could be a simulation of a robot or vehicle, and the reinforcement learning system could be trained on the simulation and then, once trained, used in the real world.

[0120] In some implementations, the environment is a real-world manufacturing environment used to manufacture products such as chemical, biological, or mechanical products, or food. As used herein, "manufacturing" a product also includes refining starting materials to create the product, or processing starting materials, for example, to remove contaminants, to produce a cleaned or recycled product. A manufacturing plant may include multiple manufacturing units, such as containers for chemical or biological substances, or machines for handling solids or other materials, such as robots. Manufacturing units are configured such that intermediate versions or components of the product can move between manufacturing units during product manufacturing, for example, via piping or mechanical transport. As used herein, product manufacturing also includes food manufacturing by kitchen robots.

[0121] Intelligent agents can include electronic intelligent agents configured to control manufacturing units or machines, such as robots, to manufacture products. That is, intelligent agents can include control systems configured to control the manufacture of chemical, biological, or mechanical products. For example, a control system may be configured to control one or more manufacturing units or machines, or to control the movement of intermediate versions or components of a product between manufacturing units or machines.

[0122] As an example, tasks performed by an agent may include manufacturing a product or an intermediate version or component thereof. As another example, tasks performed by an agent may include controlling (e.g., minimizing) the use of resources, such as controlling electricity or water consumption or the consumption of any materials or consumables used in the manufacturing process.

[0123] Actions can include control actions used to control the use of machines or manufacturing cells that handle solid or liquid materials to manufacture a product or its intermediate products or components, or to control the movement of intermediate versions of a product or components within a manufacturing environment, such as between manufacturing cells or machines. Generally, an action can be any action that affects the state of an observed environment, for example, an action configured to adjust any of the sensed parameters described below. These actions can include actions to adjust the physical or chemical conditions of a manufacturing cell, or actions to control the movement of mechanical components of a machine or joints of a robot. Actions can include actions that impose operating conditions on a manufacturing cell or machine, or actions that result in changes to settings to adjust, control, or turn the operation of the manufacturing cell or machine on or off.

[0124] Rewards or rewards can be related to metrics of task performance. For example, in the case of a task involving the manufacture of a product, the metric could include the quantity of the product manufactured, the quality of the product, the rate at which the product was manufactured, or a metric of the physical costs of performing the manufacturing task, such as the amount of energy, materials, or other resources used to perform the task. In the case of a task involving the control of resource usage, the metric could include any metric of resource usage.

[0125] Generally, observations of the state of the environment can include any electronic signals representing the function of electronic and / or mechanical items of the equipment. For example, a representation of the state of the environment can be derived from observations made by sensors that sense the state of the manufacturing environment (e.g., sensors that sense the state or configuration of a manufacturing cell or machine, or sensors that sense the movement of materials between manufacturing cells or machines). As some examples, such sensors can be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; the quantity, level, flow / movement rate, or flow / movement path of one or more materials; physical or chemical conditions, such as physical state, shape, or configuration, or chemical state such as pH; the configuration of a cell or machine (such as the mechanical configuration of a cell or machine), or valve configuration; image or video sensors for capturing image or video observations of the manufacturing cell or machine or its movement; or any other suitable type of sensor. In the case of a machine such as a robot, observations from sensors can include observations of the position, linear or angular velocity, force, torque or acceleration, or attitude of one or more parts of the machine, for example, data characterizing the current state of the machine or robot or an item held or handled by the machine or robot. Observations can also include, for example, sensed electronic signals, such as motor current or temperature signals, or image or video data, such as from a camera or LiDAR sensor. Sensors like these can be part of an agent in the environment or located separately from the agent.

[0126] In some implementations, the environment is a real-world environment comprising service facilities for multiple electronic devices, such as server clusters or data centers, like telecommunications data centers, or computer data centers for storing or processing data, or any service facility. The service facility may also include auxiliary control equipment for controlling the operating environment of the various devices, such as environmental control equipment, such as temperature control (e.g., cooling) equipment, or airflow control or air conditioning equipment. Tasks may include tasks for controlling (e.g., minimizing) the use of resources, such as tasks for controlling power consumption or water consumption. Intelligent agents may include electronic intelligent agents configured to control the operation of the devices or to control the operation of auxiliary (e.g., environmental) control equipment.

[0127] Generally, an action can be any action that affects the state of the observed environment, such as an action configured to adjust any of the sensed parameters described below. These actions can include actions that apply operating conditions to control equipment or auxiliary control equipment, such as actions that cause changes in settings to adjust, control, or turn the equipment or auxiliary control equipment on or off.

[0128] Generally, observations of the state of the environment can include any electronic signals indicating the operation of a facility or equipment within a facility. For example, an indication of the state of the environment can be derived from observations made by any sensor sensing the state of the physical environment of the facility or by any sensor sensing the state of one or more pieces of equipment or one or more auxiliary control devices within the facility. These sensors include those configured to sense: electrical conditions, such as current, voltage, power, or energy; the temperature of the facility; the flow rate, temperature, or pressure of fluids within the facility or its cooling system; or the configuration of the physical facility, such as whether vents are open.

[0129] Rewards or rewards can be related to metrics of task performance. For example, in the case of tasks that control (e.g., minimize) the use of resources, such as controlling the use of electricity or water, the metric can include any measure of resource usage.

[0130] In some implementations, the environment is the real-world environment of the power generation facility (e.g., a renewable power generation facility such as a solar power plant or a wind power plant). Tasks may include control tasks that control the electricity generated by the facility (e.g., control the delivery of electricity to a distribution network), such as to meet demand or reduce the risk of mismatch between grid components, or to maximize the electricity generated by the facility. Agents may include electronic agents configured to control the generation of electricity by the facility or the coupling of the generated electricity to the grid. Actions may include actions for controlling the electrical or mechanical configuration of a generator (such as the electrical or mechanical configuration of one or more renewable power generation elements), for example, to control the configuration of a wind turbine or one or more solar panels or mirrors, or the electrical or mechanical configuration of a rotating generator. Mechanical control actions may, for example, include actions that control the conversion of energy input to electrical output (e.g., the efficiency of the conversion) or the degree of coupling of energy input to electrical output. Electrical control actions may, for example, include actions that control one or more of the voltage, current, frequency, or phase of the generated electricity.

[0131] Rewards or rewards can be related to metrics of task performance. For example, in the case of a task controlling the delivery of electricity to the distribution network, the metric could be related to a measure of the electricity delivered, or a measure of electrical mismatches (such as voltage, current, frequency, or phase mismatches) between the generating facility and the grid, or a measure of power or energy losses in the generating facility. In the case of a task maximizing the delivery of electricity to the distribution network, the metric could be related to a measure of the electricity or energy delivered to the grid, or a measure of power or energy losses in the generating facility.

[0132] Generally, observations of the state of the environment can include any electronic signals representing the electrical or mechanical operation of the power generation equipment in a power generation facility. For example, a representation of the state of the environment can be derived from observations by any sensor that senses the physical or electrical state of the equipment in a power generation facility that is generating electricity, or the physical environment of such equipment, or the state of auxiliary equipment supporting the power generation equipment. Such sensors can include those configured to sense: the electrical state of the equipment, such as current, voltage, power, or energy; the temperature or cooling of the physical environment; fluid flow; or the physical configuration of the equipment; and observations of the electrical state of the power grid, such as from local or remote sensors. Observations of the state of the environment can also include one or more predictions about the future operating conditions of the power generation equipment, such as predictions of future wind speeds or solar irradiance, or predictions of the future electrical state of the power grid.

[0133] As another example, the environment can be a chemical synthesis or protein folding environment, such that each state is a corresponding state of a protein chain or one or more intermediate or precursor chemicals, and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, actions are possible folding actions for folding the protein chain or actions for assembling the precursor chemical / intermediate, and the desired outcome can include, for example, folding the protein to make it stable and enabling it to perform a specific biological function, or providing an efficient synthetic route for the chemical. As another example, the agent can be a mechanical agent that executes or controls protein folding actions or chemical synthesis steps automatically selected by the system without human intervention. Observations can include direct or indirect observation of the state of the protein or chemical / intermediate / precursor, and / or can be derived from a simulation.

[0134] In a similar manner, the environment can be a drug design environment, such that each state is the corresponding state of a potential pharmaceutically active compound, and the agent is a computer system for determining the elements of the pharmaceutically active compound and / or the synthetic pathway for the pharmaceutically active compound. For example, in a simulation, a drug / synthesis can be designed based on a reward derived from the drug's target. As another example, the agent can be a mechanical agent that performs or controls the synthesis of a drug.

[0135] In some further applications, the environment is a real-world environment, and the agent manages the allocation of tasks across computing resources (e.g., on mobile devices and / or in data centers). In these implementations, actions may include assigning tasks to specific computing resources.

[0136] As a further example, actions may include presenting an advertisement, observations may include ad impressions or click counts or click-through rates, and rewards may represent previous selections of items or content by one or more users.

[0137] In some cases, observations may include textual or verbal instructions provided to the agent by a third party (e.g., the agent's operator). For example, the agent may be an autonomous vehicle, and the user of the autonomous vehicle may provide textual or verbal instructions to the agent (e.g., to navigate to a specific location).

[0138] As another example, the environment can be an electromechanical or mechatronic design environment, such as an environment in which the design of an electromechanical or mechatronic entity is simulated. The simulated environment can be a simulation of a real-world environment in which the entity is intended to operate. The task can be designing the entity. Observations can include observations characterizing the entity, i.e., observations of the entity's mechanical shape or electromechanical or mechatronic configuration, or observations of the entity's parameters or properties. Actions can include actions that modify the entity, such as actions that modify one or more of the observations. Rewards or rewards can include one or more metrics of the performance of the entity's design. For example, rewards or rewards can be related to one or more physical characteristics of the entity (such as weight or force); or to one or more electrical characteristics of the entity (such as a measure of efficiency in performing a specific function designed for the entity). The design process can include outputting a design for manufacturing, for example, in the form of computer-executable instructions for manufacturing the entity. The process can include manufacturing the entity according to the design. Thus, the design of the entity can be optimized, for example, through reinforcement learning, and then the optimized design is output for manufacturing the entity, for example, as computer-executable instructions; the entity with the optimized design can then be manufactured.

[0139] As previously mentioned, the environment can be a simulated environment. Generally, in the case of a simulated environment, observations can include simulated versions of one or more of the previously described observations or observation types, and actions can include simulated versions of one or more of the previously described actions or action types. For example, the simulated environment can be a motion simulation environment (e.g., a driving simulation or a flight simulation), and the agent can be a simulated vehicle navigating through the motion simulation. In these implementations, actions can be control inputs for controlling the simulated user or the simulated vehicle. Generally, the agent can be implemented as one or more computers interacting with the simulated environment.

[0140] The simulated environment can be a simulation of a specific real-world environment and an intelligent agent. For example, the system can be used to select actions in the simulated environment during training or evaluation, and after training or evaluation, or both, can be deployed to control a real-world agent in the specific real-world environment that is the object of the simulation. This avoids unnecessary depletion and damage to the real-world environment or the real-world agent, and allows the control neural network to be trained and evaluated in situations where it is rare, difficult, or unsafe to recreate in the real-world environment. For example, the system can be partially trained using a simulation of a mechanical agent in a simulation of a specific real-world environment, and then deployed to control a real mechanical agent in that specific real-world environment. Therefore, in such cases, the observation of the simulated environment is relevant to the real-world environment, and the selected actions in the simulated environment are relevant to the actions to be performed by the mechanical agent in the real-world environment.

[0141] Optionally, in any of the above implementations, the observation at any given time step may include data from previous time steps that can be helpful in characterizing the environment, such as actions performed at previous time steps, rewards received at previous time steps, or both.

[0142] The term "configured" is used in this specification to combine system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon that causes the system to perform those operations or actions in operation. For one or more computer programs configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform that operation or action.

[0143] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals—e.g., machine-generated electrical, optical, or electromagnetic signals—to generate artificially generated propagation signals to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0144] The term "data processing device" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0145] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored as a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a data communication network.

[0146] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0147] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a dedicated logic circuit system (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmable computers.

[0148] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to or from them, or both. However, a computer does not necessarily have to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0149] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0150] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Similarly, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in response.

[0151] Data processing devices used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and computationally intensive portions of machine learning training or production (i.e., inference, workloads).

[0152] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0153] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components (e.g., as a data server), or middleware components (e.g., an application server), or frontend components (e.g., a client computer having a graphical user interface, web browser, or app that a user can interact with through an implementation of the subject matter described in this specification), or any combination of one or more such backend, middleware, or frontend components. Components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0154] A computing system may include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship is established by computer programs executed on respective computers that establish a client-server relationship between them. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device (e.g., the result of user interaction) can be received from that device at the server.

[0155] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in this specification in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may be removed from the combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

[0156] Similarly, although operations are depicted in the accompanying drawings and listed in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in an ordered sequence, or requiring all shown operations to be performed to achieve the desired result. In some contexts, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0157] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method for training a high-level controller neural network, executed by one or more computers, the high-level controller neural network being configured to receive input including observations characterizing the state of an environment with which an agent is interacting, and to generate output defining natural language commands for a low-level controller neural network, the low-level controller neural network generating control outputs for controlling the agent, the method comprising: Obtain the training dataset, which includes: Multiple demonstration trajectories, for each of multiple time steps, each demonstration trajectory includes a corresponding observation characterizing the state of the demonstration environment with which the demonstration agent is interacting at that time step, and a corresponding natural language command provided to the demonstration agent at that time step; and The advanced controller neural network is trained on the demonstration trajectory in the training dataset using supervised learning.

2. The method of claim 1, wherein, Training the high-level controller neural network on the demonstration trajectories in the training dataset via supervised learning includes: The high-level controller neural network is trained on the demonstration trajectories in the training dataset to minimize the behavior cloning loss, which measures the probability, for each of the plurality of time steps in each demonstration trajectory, of being assigned to the corresponding natural language command provided to the demonstration agent at that time step by means of the output generated by the high-level controller neural network through processing inputs including the corresponding observations for that time step.

3. The method of any preceding claim, wherein, The training further includes: Generate reinforcement learning trajectories, wherein the generation includes: At each of the multiple first time steps in the reinforcement learning trajectory: Receive current observations representing the state of the training environment with which the training agent is interacting; The advanced controller neural network is used to process the input, including the current observation, to generate an output that defines a natural language command for the first time step; The low-level controller neural network is used to process the input, including the natural language command for the first time step, to generate a control output for the first time step; The control output is used to control the trained agent; and Receive the reward from the first time step; and The high-level controller neural network is trained by reinforcement learning using at least the reward from the first time step.

4. A method for training a high-level controller neural network, executed by one or more computers, the high-level controller neural network being configured to receive input including observations characterizing the state of an environment with which an agent is interacting, and to generate output defining natural language commands for a low-level controller neural network, the low-level controller neural network generating control outputs for controlling the agent, the method comprising: Generate reinforcement learning trajectories, wherein the generation includes: At each of the multiple first time steps in the reinforcement learning trajectory: Receive current observations representing the state of the training environment with which the training agent is interacting; The advanced controller neural network is used to process the input, including the current observation, to generate an output that defines a natural language command for the first time step; The low-level controller neural network is used to process the input including the natural language command for the first time step to generate a control output for the first time step; The control output is used to control the trained agent; and Receive the reward from the first time step; and The high-level controller neural network is trained by reinforcement learning using at least the reward from the first time step.

5. The method of claim 4, further comprising: Obtain the training dataset, which includes: Multiple demonstration trajectories, for each of the multiple time steps, each demonstration trajectory includes a corresponding observation characterizing the state of the demonstration environment with which the demonstration agent is interacting at that time step, and a corresponding natural language command provided to the demonstration agent at that time step; and The advanced controller neural network is trained on the demonstration trajectory in the training dataset using supervised learning.

6. The method of any one of claims 3 to 5, wherein, The input to the low-level controller neural network further includes the current observation at the first time step.

7. The method of any one of claims 3 to 6, wherein the generation comprises: At each of the multiple second time steps in the reinforcement learning trajectory: Receive current observations representing the state of the training environment with which the training agent is interacting; It was determined that the criteria for generating new natural language commands were not met; and In response: The low-level controller neural network is used to process the input, including the natural language command from the most recent first time step, to generate a control output for the second time step; The control output is used to control the trained agent; as well as Receive the reward for the second time step.

8. The method of claim 7, wherein, Training the high-level controller neural network through reinforcement learning includes: The high-level controller neural network is trained by reinforcement learning using at least the rewards from the first and second time steps.

9. The method as described in any of the preceding claims, wherein, The low-level controller neural network has been pre-trained prior to the training of the high-level controller neural network and remains fixed during the training of the high-level controller neural network.

10. The method of claim 9, wherein, The low-level controller neural network has been pre-trained through supervised learning on multiple low-level demonstration trajectories. For each of the multiple time steps, each low-level demonstration trajectory includes a corresponding observation representing the state of the demonstration environment with which the demonstration agent is interacting at that time step, a corresponding natural language command provided to the demonstration agent at that time step, and an action performed by the demonstration agent at that time step.

11. The method of claim 9, wherein, The low-level controller has been pre-trained on the multiple low-level demonstration trajectories through behavior cloning.

12. The method as described in any of the preceding claims, wherein, At each time step, the advanced controller neural network is conditioned on observations from one or more previous time steps.

13. The method of claim 12, wherein, The advanced controller neural network includes one or more recurrent layers.

14. The method of claim 12 or claim 13, wherein, The advanced controller neural network includes one or more self-attention layers.

15. The method as claimed in any of the preceding claims, wherein, The environment referred to is either the simulated environment or the real-world environment.

16. The method of claim 15, wherein, The environment is a real-world environment, and the observation is obtained from one or more sensors that sense the real-world environment.

17. The method of claim 16, wherein, The intelligent agent is a mechanical robot that interacts with the real-world environment.

18. The method as claimed in any of the preceding claims, wherein, The demonstration environment is the same as the environment described above.

19. The method according to any one of claims 1 to 17, wherein, The demonstration environment is different from the environment described above.

20. The method as described in any of the preceding claims when it is dependent on claim 3 or claim 4, wherein, The training environment is the same as the environment described above.

21. The method as claimed in any one of claims 1 to 19 when it is subordinate to claim 3 or claim 4, wherein, The training environment is different from the environment described above.

22. The method as described in any of the preceding claims when it is dependent on claim 1 or claim 5, wherein, The demonstration agent is controlled by a user or by expert policies.

23. The method as described in any of the preceding claims when it is subordinate to claim 5, wherein, Training the high-level controller neural network on the demonstration trajectories in the training dataset via supervised learning includes: training the high-level controller neural network to minimize a behavioral cloning loss, the behavioral cloning loss measuring, for each of the plurality of time steps in each demonstration trajectory, the probability that a natural language command is assigned to the demonstration agent at that time step by means of an output generated by the high-level controller neural network through processing an input including the corresponding observation for that time step.

24. The method as described in any of the preceding claims, wherein, The environment is a video game environment, and the agent is an agent within the video game environment.

25. A system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform operations of a corresponding method as claimed in any one of claims 1 to 24.

26. A computer storage medium for storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of a corresponding method according to any one of claims 1 to 24.

27. An intelligent agent including a control system, the control system comprising: An advanced controller neural network, which is trained by the method according to any one of claims 1 to 22 and configured to receive input including observations characterizing the state of the environment with which the agent is interacting and to generate outputs defining natural language commands; as well as A low-level controller neural network is configured to process the natural language commands and generate control outputs for controlling the agent.

Images