Training Multimodal Interactive Agents with Reward Models

Abstract

A method, system, and apparatus for controlling an agent, including a computer program encoded on a computer storage medium. In particular, the interactive agent can be controlled by a neural network trained with reward values ​​using reinforcement learning.

Description

[Technical Field]

[0001] This specification relates to controlling agents using neural networks.

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the input of the current value of its respective parameter set. Summary of the Invention [Means for solving the problem]

[0003] This specification describes a system, implemented as a computer program on one or more computers in one or more locations, that controls interactive agents interacting in an environment by selecting actions to be performed by the agents and then causing the agents to perform the actions.

[0004] The agent is referred to as an "interactive" agent because, as part of interacting with the environment, the agent interacts with one or more other agents in the environment. The one or more other agents may include humans, other agents controlled by different computer systems, or both. An interactive agent interacts with other agent(s) by receiving communications generated by the other agent(s) and, optionally, by generating text that is communicated to the other agent(s).

[0005] In particular, interactions with other agents provide an interactive agent with information about what tasks the agent needs to perform in the environment at any given time.

[0006] In one aspect, a method executed by one or more computers for training a policy neural network to control an agent interacting with an environment includes, at each of a plurality of time steps, obtaining observations for the time step, the observations including an image characterizing a state of the environment at the time step and a natural language text sequence for the time step characterizing a task being performed by the agent in the environment at the time step, the method further includes, at each of the plurality of time steps, processing a policy input including the observation image and the natural language text sequence using the policy neural network to select one or more actions to be performed by the agent in response to the observation image; and processing a reward input including the observation image and the natural language text sequence using a reward neural network, the reward neural network configured to process the observation image and the natural language text to generate a reward output defining a reward value that characterizes progress of the agent in performing the task characterized by the natural language text sequence at the time step, the method further includes training the policy neural network by reinforcement learning using the respective reward values ​​for the plurality of time steps.

[0007] In some embodiments, prior to training the policy neural network by reinforcement learning, the method includes training the policy neural network by imitation learning on data characterizing interactions between multiple agents in an environment.

[0008] In some implementations, the reward neural network is configured to generate a reward value that represents the utility for performing a task of the trajectory of the observation to the observation for the time step.

[0009] In some implementations, training the policy neural network by reinforcement learning using the respective reward values ​​for the plurality of time steps includes: calculating, for each of the plurality of time steps, a per-time-step reward based on a difference between the reward value at the time step and the reward value for the preceding time step in the sequence; and training the policy neural network by reinforcement learning using the respective per-time-step rewards for the plurality of time steps.

[0010] In some implementations, the reward input includes the reward value generated for the previous time step in the sequence.

[0011] In some implementations, the reward input further includes a natural language output generated by the agent in a previous time step.

[0012] In some implementations, the reward output is a reward value.

[0013] In some implementations, the reward output comprises a probability distribution over a set of possible reward values.

[0014] In some embodiments, the reward neural network is trained with training data that includes, for each of one or more time steps within each of a plurality of training task episodes, a respective reward label selected from a set of reward labels that include: a negative reward label that indicates, at the time of the time step, that the agent has regressed from achieving a goal characterized in the natural language instructions for the time step; and a positive reward label that indicates, at the time step, that the agent has made progress in achieving a goal characterized in the natural language instructions for the time step.

[0015] In some implementations, the reward neural network is trained based on training data that includes, for each of one or more time steps within each of a plurality of training task episodes, a respective reward label selected from a set of reward labels that also includes a neutral reward label.

[0016] In some implementations, the reward model is trained on the training data with a loss function that measures the difference between predicted reward values ​​for two time steps within the same training task episode.

[0017] In some implementations, the reward neural network is trained on the training data and for a given pair of time steps within the same training task episode, where if both time steps in the pair have positive reward labels and no intervening time steps in the pair have negative reward labels, the loss function encourages (e.g., biases or tends to encourage) the predicted reward value for the later time step in the pair to be larger than the reward value predicted for the earlier time step in the pair; and if both time steps in the pair have negative reward labels and no intervening time steps in the pair have positive reward labels, the loss function encourages the predicted reward value for the earlier time step in the pair to be larger than the reward value predicted for the later time step in the pair.

[0018] In some implementations, the reward neural network is trained on training data that includes a given pair of time steps, where both time steps in the pair have a neutral reward label and all time steps between the time steps in the pair also have a neutral reward label, and the loss function encourages the predicted reward value for the later time step in the pair to be equal to the predicted reward value for the earlier time step in the pair.

[0019] In some implementations, the loss function measures, for each time step within each of the training episodes, the error between the reward label for the time step and the reward value for the time step.

[0020] In some implementations, processing a reward input including an observed image and a natural language text sequence using a reward neural network includes using an image embedding neural network to generate a plurality of image embeddings representing the observed image, processing the input including the natural language text sequence using a text embedding neural network to generate a plurality of text embeddings representing at least the natural language text sequence, processing the input including the image embeddings and the text embeddings using a multimodal neural network to generate aggregate embeddings, and processing the input including the aggregate embeddings using a reward neural network head to generate a reward value. That is, the reward neural network head may include one or more neural network layers having an output layer, the one or more neural network layers configured to process the input including the aggregate embeddings such that the output layer generates the reward value.

[0021] In some implementations, the multimodal neural network is a multimodal Transformer neural network configured to (i) apply self-attention across at least the text embeddings and the image embeddings to generate updated embeddings for at least each of the plurality of text embeddings, and (ii) generate an aggregate embedding from the updated embeddings for at least each of the text embeddings.

[0022] In some implementations, the multimodal Transformer neural network includes one or more self-attention layers, each having one or more self-attention heads, and applying self-attention includes processing the input through the one or more self-attention layers.

[0023] In some implementations, inputs to the multimodal Transformer neural network include image embeddings, text embeddings, and one or more dedicated embeddings (e.g., embeddings that do not depend on the observations at the time steps).

[0024] In some implementations, applying self-attention includes generating updated embeddings for each of the text embedding and the private embedding without updating the image embedding.

[0025] In some implementations, each self-attention head of each self-attention layer is configured to (i) receive head input including image embeddings generated by the image embedding neural network and current embeddings for each of the text embeddings and dedicated embeddings; (ii) generate, from the current embeddings, respective queries corresponding to each of the text embeddings and dedicated embeddings; (iii) generate, from the image embeddings and the current embeddings, respective keys corresponding to each of the image embeddings, text embeddings, and dedicated embeddings; (iv) generate, from the image embeddings and the current embeddings, respective values ​​corresponding to each of the image embeddings, text embeddings, and dedicated embeddings; and (v) apply query-key-value attention across the respective queries, keys, and values ​​to generate respective initial updated embeddings for each of the text embeddings and dedicated embeddings without updating the image embeddings.

[0026] In some implementations, generating the aggregated embeddings includes aggregating updated embeddings for each of the text embeddings and the dedicated embeddings to generate an initial aggregated embedding, and combining the updated embeddings for each of the dedicated embeddings with the initial aggregated embedding to generate the aggregated embedding.

[0027] In some implementations, combining includes concatenating each updated embedding with each of the initial aggregate embeddings.

[0028] In some implementations, processing the input including the aggregate embedding using a reward neural network head to generate a reward output includes generating a state representation from the aggregate embedding, and processing the state representation using one or more neural network layers to generate the reward output.

[0029] In some implementations, generating the state representation includes processing the state representation using a memory neural network.

[0030] In some implementations, the memory neural network is a recurrent neural network.

[0031] In some implementations, the reward input further comprises a natural language output produced by the agent in a previous time step, the input to the text embedding neural network further comprises the natural language output, and the text embedding represents the natural language text sequence and the natural language output.

[0032] In some implementations, the natural language text sequences are generated from natural language text sequences that are generated based on corresponding natural language text sequences from corresponding time steps within the training task episode.

[0033] In some implementations, the natural language text sequences are generated by a setter agent within the environment.

[0034] In some embodiments, the setter agent is controlled using a setter neural network.

[0035] In some implementations, the setter neural network is trained to mimic an expert setter agent through imitation learning.

[0036] In some implementations, the reward neural network is trained over a loss function that includes (i) a loss function that is based on the reward label and (ii) one or more auxiliary losses.

[0037] In some implementations, the one or more auxiliary losses include an imitation learning loss calculated using outputs generated by an auxiliary policy neural network head that generates policy outputs for controlling the agent.

[0038] In some implementations, the one or more auxiliary losses include a control self-supervised representation learning loss.

[0039] In some implementations, the one or more auxiliary losses include a cross-modality matching loss that uses outputs generated by a multimodal neural network.

[0040] In some implementations, at least some of the parameter values ​​of the reward neural network were initialized using parameter values ​​of the policy neural network determined by training the policy neural network by imitation learning.

[0041] In some embodiments, training the policy neural network by reinforcement learning using the respective reward values ​​for the plurality of time steps includes training the policy neural network by reinforcement learning and by imitation learning on an imitation learning dataset.

[0042] In some implementations, each observation is captured by a camera sensor on the agent or a camera sensor located within the environment.

[0043] In some implementations, the agent is a machine agent that interacts with a real-world environment to achieve a specified goal (e.g., specified by natural language input) by performing actions selected by a trained policy neural network in response to observations of the real-world environment.

[0044] In some implementations, the agent is a software agent configured to control electromechanical devices within the real-world environment to achieve a specified goal (e.g., specified by natural language input) by performing actions selected by a trained policy neural network in response to observations of the real-world environment.

[0045] In some implementations, the environment is a computing environment and the agent is a software agent executing within the computing environment to control one or more computing devices by performing actions selected by a trained policy neural network to perform a task specified by a user interacting with the software agent. For example, a user may specify a task via natural language input, which may be provided by the user in the form of speech and / or text.

[0046] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0047] Generally, this document describes techniques for training neural networks to control interactive agents to perform tasks specified by natural language instructions issued by other agents in an environment.

[0048] Controlling such agents can be particularly useful in many real-world tasks that require the agent to perform a variety of tasks that are not fixed in advance, but are described by statements made by other agents, e.g., agents controlled using human or other strategies. However, training data for these types of tasks can be difficult to collect, and the open-ended nature of the tasks fails to encompass the wide variety of tasks that the agent can be commanded to perform after training.

[0049] To enable a system to effectively control an agent despite these challenges, this specification describes various techniques that can be used together or separately to improve interactive agent control and to enable a system to effectively control an agent to perform new tasks not seen in the training data.

[0050] As an example, this specification describes a neural network (a "perceptual encoder") that effectively combines multimodal inputs to generate an encoded representation of the environment, enabling the system to generate an accurate representation of the environmental state, resulting in more precise control and better generalization to new tasks.

[0051] As another example, this specification describes a reward neural network that is trained based on expert evaluations of agent interactions in an environment to determine whether the agent is performing actions that advance toward a presented objective. The reward neural network is trained to output reward values ​​that are used to train a policy neural network using reinforcement learning techniques. For example, the policy neural network can be first trained using imitation learning and then fine-tuned using a reinforcement learning approach with the output of the reward neural network. By utilizing the reward neural network, the system can accurately estimate rewards for a wide range of tasks, rather than being limited to tasks already performed by the expert agent. Thus, the trained policy neural network can better generalize to new tasks after training.

[0052] For example, an agent trained using only imitation learning may be able to reach a basic level of competency. However, by being initially trained by imitation learning and then fine-tuning, an agent trained by reinforcement learning can reach human competency levels for many tasks. As explained in more detail below, improvements such as these apply to both interactions involving mobile manipulation (environmental locomotion and sophisticated physical interaction) and tasks involving two-way linguistic interaction, such as question-answering.

[0053] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0054] [Figure 1] 1 illustrates an exemplary action selection system. [Figure 2] FIG. 1 is a flow diagram of an exemplary process for controlling an agent at a time step. [Figure 3] FIG. 1 is a flow diagram of an example process for generating an encoded representation at a time step. [Figure 4] FIG. 1 is a flow diagram of an example process for generating a sequence of actions. [Figure 5] FIG. 1 is a flow diagram of an exemplary process for training a policy network through reinforcement learning. [Figure 6] FIG. 1 is a flow diagram of an exemplary process for generating training data for training a reward neural network. [Figure 7] FIG. 1 is a flow diagram of an exemplary process for training a reward neural network. [Figure 8] FIG. 1 is a flow diagram of an exemplary process for training a policy network through reinforcement learning. [Figure 9] FIG. 1 is a flow diagram of an exemplary process for generating reward values ​​from a reward neural network. [Figure 10] 1 shows the improvement of an agent directed by a policy neural network with reinforcement learning compared to a policy neural network with imitation learning alone.Like reference symbols and designations in the various drawings refer to like elements. [Figure 11] 1 shows a diagram of an exemplary process for improving a policy neural network through reinforcement learning using the output of a reward model. [Figure 12] 1 shows a diagram of an example process for training a policy neural network, training a reward neural network, and training the policy neural network with the reward neural network using reinforcement learning. [Figure 13] 1 shows an example of data collected to train a reward neural network. [Figure 14]1 shows an exemplary algorithm for implementing the "Inter-temporal Bradley-Terry" model for training a reward neural network. DETAILED DESCRIPTION OF THE INVENTION

[0055] 1 illustrates an exemplary action selection system 100. Action selection system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, in which the systems, components, and techniques described below are implemented.

[0056] The action selection system 100 controls an interactive agent 104 interacting within an environment 106 by selecting an action 108 to be performed by the agent 104 and then causing the agent 104 to perform the action 108 .

[0057] In particular, at each time step, the system 100 receives multimodal input, e.g., input including data of multiple modalities (e.g., data types), that includes at least an observed image 110 that characterizes the state of the environment at the time step and a natural language text sequence 130, and uses the multimodal input to control the agent 104.

[0058] This agent 104 is referred to as an "interactive" agent because, as part of interacting with the environment, it interacts with one or more other agents 107 in the environment. The one or more other agents 107 may include humans, other agents controlled by different computer systems, or both. The interactive agent 104 interacts with the other agent(s) 107 by receiving communications generated by the other agent(s) 107 and, optionally, by generating text that is communicated to the other agent(s) 107.

[0059] In particular, at each time step, the system 100 receives an observed image 110 that characterizes the state of the environment 106 at the time step. The observed image 110 can be captured, for example, by a camera sensor on the agent 104 or by another camera sensor located within the environment 106.

[0060] The system 100 also receives, for each time step, a natural language text sequence 130. In particular, the natural language text sequence 130 can be the result of a communication from one of the other agents 107 in the environment 106. For example, the other agent 107 can speak an utterance (e.g., verbalize or express an audible utterance), and the interactive agent 104 or the system 100 can transcribe the utterance to generate the natural language text sequence 130.

[0061] In some cases, there may not be new communication at every time step. In these cases, the system 100 may use the most recently received natural language text sequence as the time step's text sequence 130. That is, the system 100 may reuse the most recently received natural language text sequence until a new text sequence is received.

[0062] In general, the natural language text sequence 130 provides information about the task that the agent 104 needs to perform within the environment 106, e.g., information about the goal that the agent 104 needs to attempt to reach by acting in a time step. However, because the text is natural language text that may be generated by other agents 107, the content of the text may underspecify the goal, may be ambiguous as to which goal needs to be reached, or may require clarification. That is, the text sequence 130 may provide insufficient information for the agent 104 to perform the intended task.

[0063] The system 100 then processes the policy input, which includes at least the observed image 110 and the natural language text sequence 130, using a policy neural network to select one or more actions 108 to be performed by the agent in response to the observation, e.g., in response to the image 110 and the sequence 130.

[0064] In particular, in some embodiments, the agent 104 performs a single action 108 in response to each observed image 110, such that, for example, a new observed image 110 is captured after each action performed by the agent. In these embodiments, the system 100 causes the agent 104 to perform a single action, for example, by providing instructions to the agent 104, which, when executed, cause the agent to perform the single action by submitting control inputs directly to an appropriate control of the agent 104, by providing data identifying the action to a control system of the agent 104, or using other suitable control techniques.

[0065] In some other implementations, the agent 104 performs a sequence of actions 108 in response to each observed image 110, such that, for example, the actions are performed by the agent before the next observed image 110 is captured. In these implementations, the system 100 generates a sequence of actions 108 including respective actions 108 at multiple locations and causes the agent 104 to perform the sequence of actions 108 in sequence order, e.g., by first performing the action at a first location, then performing the action at a second location, etc. The system 100 can cause the agent 104 to perform a given action as described above.

[0066] The policy neural network may generally have any suitable architecture that enables the policy neural network to map received observations, including both images and text, to outputs that define one or more actions to be performed by the agent 104.

[0067] For example, the policy neural network may include a perceptual encoder neural network 122 that generates an encoded representation 124 for a time step, and the system may then process the encoded representation 124 to generate an output that defines one or more actions 108.

[0068] For example, system 100 may generate one or more actions 108 at a given time step by generating a state representation from encoded representation 124 (also referred to below as an "aggregate embedding") and then processing the state representation using a policy sub-neural network 126 of the policy neural network. Because the state representation can be the same as encoded representation 124 or can be generated by a memory neural network, e.g., a recurrent neural network, of the policy neural network, the state representation can incorporate information from previous environmental states.

[0069] In some implementations, in addition to or instead of performing one or more actions 108 in response to observed images 110, system 100 may use encoded representations 124 at some or all of the time steps to generate and provide as output output text sequences. In particular, system 100 may process input derived from encoded representations 124, e.g., input including state representations, using a natural language generation neural network of a policy neural network to generate output text sequences at the time steps.

[0070] The system 100 may then generate audio representing the output text sequence, which may be played by the interactive agent 104 or otherwise communicated to other agent(s) 107 in the environment for the agent 104 to interact with the other agent(s) 107. Interacting with the other agents 107 by generating text, audio, or both may enable the interactive agent 104 to ask questions of the other agent(s) 107 or otherwise obtain additional information from the other agent(s) 107 about how to perform a desired task, for example, by prompting the other agent(s) 107 to provide additional information.

[0071] Processing the input observations to generate action(s) and optionally output text sequences is described in further detail below with reference to FIGS.

[0072] Before using the policy neural network to control an agent, a neural network training system 190 uses a reinforcement learning reward neural network 192 to train the policy neural network, for example, to determine trained values ​​for the parameters of the policy neural network.

[0073] In some implementations, before training the policy neural network by reinforcement learning, system 190 trains the neural network by imitation learning, e.g., with ground truth data generated by an expert agent. The ground truth data includes a set of ground truth trajectories, each including observations having observed images and natural language text sequences in a sequence of time steps, and one or more of ground truth actions and ground truth text outputs. A "ground truth" action is a target action that needs to be performed by the agent at a given time step (or a given position in an action sequence). Similarly, a "ground truth" text output is a target text output that needs to be generated by the system at a given time step. For example, the ground truth action and text output can be the actual action and text output performed or generated (e.g., spoken) by the expert agent (respectively) at a given time step. The expert agent can be, for example, an agent controlled by a human user, an agent controlled by a previously learned policy, or an agent controlled by a hard-coded, heuristic-based policy.

[0074] Reward neural network 192 is a neural network configured to process the observed images and natural language text during a given time step to generate a reward output that defines a reward value that characterizes the agent's progress in performing a task characterized by the natural language text sequence at the time step.

[0075] For example, the reward output can be a reward value, i.e., the neural network 192 regresses toward a reward value. As another example, the reward output can be a probability distribution over a set of possible reward values. In this example, the system 190 can use the probability distribution to sample, or greedily select, one of the reward values.

[0076] For example, during reinforcement learning training, system 190 can use reward neural network 192 to generate rewards at each time step in response to the action(s) performed in response to the observations at the time step, the text sequence generated as output at the time step, or both, and can use those rewards to train the policy neural network using off-policy reinforcement learning techniques. In some implementations, neural network training system 190 is separate from action selection system 100, i.e., action selection system 100 need not include neural network training system 190.

[0077] Training using reinforcement learning is described in more detail below.

[0078] As noted above, in some embodiments, system 100 first trains policy neural network 126 by imitation learning and then trains the policy neural network by reinforcement learning. In some other embodiments, system 100 can train the policy neural network by reinforcement learning from scratch, i.e., without any pre-training, or can pre-train some or all of the policy neural network by a different technique, e.g., an unsupervised learning technique.

[0079] Optionally, in any of the above implementations, system 190 can use auxiliary control learning losses that employ cross-modality matching to improve the training of the policy neural network. For example, the system can pre-train perceptual encoder 122 using auxiliary control learning losses, or can train the policy neural network based on a loss function that includes an imitation learning loss or a reinforcement learning loss and an auxiliary control learning loss.

[0080] Cross-modality matching refers to having a classifier neural network predict, from the encoded representation of a given observation (text sequence pair), whether a given observation and a given text sequence correspond to the same time step. An observation and a text sequence "correspond" to the same time step if the text sequence was the most recently received text sequence at the time step when the observed image was captured, e.g., if the text sequence and the observed image are temporally aligned.

[0081] In some implementations, environment 106 is a real-world environment and agent 104 is a machine agent that interacts with the real-world environment, e.g., to perform one or more selected actions within the real-world environment. For example, the agent may be a robot that interacts with the environment to achieve a goal, e.g., to place an object of interest within the environment, to move the object of interest to a specified location within the environment, to physically manipulate the object of interest within the environment in a specified manner, or to navigate to a specified destination within the environment as specified by natural language input received from another agent(s), or the agent may be an autonomous or semi-autonomous land, air, or sea vehicle that navigates through the environment to a specified destination within the environment.

[0082] An action can be a control input for controlling a mechanical agent, e.g., a robot, e.g., a torque or higher level control command for a joint of the robot, or a control input for controlling an autonomous or semi-autonomous land, air, or sea vehicle, e.g., a torque or higher level control command for a control surface or other control element of the vehicle.

[0083] In other words, an action may include, for example, position, velocity, and / or force / torque / acceleration data for one or more joints of a robot or other part of a mechanical agent. Actions may additionally or alternatively include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices in the environment, the control of which affects the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, actions may include actions for controlling navigation, such as steering, and movement, such as braking and / or accelerating the vehicle.

[0084] In some embodiments, the environment is a simulated environment and the agent is implemented as one or more computer programs that interact with the simulated environment. For example, the environment can be a computer simulation of a real-world environment and the agent can be a simulated machine agent that navigates the computer simulation.

[0085] For example, the simulated environment may be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent may be a simulated vehicle that navigates the motion simulation. In these implementations, the actions may be control inputs for controlling a simulated user or a simulated vehicle. As another example, the simulated environment may be a computer simulation of a real-world environment, and the agent may be a simulated robot that interacts with the computer simulation.

[0086] In general, when the environment is a simulated environment, the actions may include simulated versions of one or more of the actions or types of actions described above.

[0087] In some cases, the system can be used to control an agent's interactions with a simulated environment, and the system can train parameters of neural networks (e.g., perceptual encoder neural network 122, policy neural network 126, and, when used, a language generation neural network) used to control the agent based on the agent's interactions with the simulated environment. After the neural networks are trained based on the agent's interactions with the simulated environment, the trained policy neural network can be used to control the real-world agent's interactions with the real-world environment, for example, to control a simulated agent within the simulated environment. By training a deep neural network based on the agent's interactions with the simulated environment (e.g., instead of the real-world environment), agent attrition can be avoided and the likelihood that the agent may damage itself or aspects of its environment by performing poorly selected actions can be reduced. In some cases, the system can be partially trained using simulation as described above and then further trained in the real-world environment.

[0088] As another example, the environment may be a video game and the agent may be an agent within the video game that interacts with one or more other agents, e.g., agents controlled by one or more human users.

[0089] As yet another example, the environment can be an augmented reality or virtual reality representation of a real-world environment, and the agent can be an entity within the representation that interacts with one or more other agents, e.g., agents controlled by one or more human users. In the case of an augmented reality environment, the observed image can include image data characterizing the real-world environment, e.g., including objects of interest within the environment. The agent can be a software agent that can be configured to control electromechanical devices within the real-world environment to perform one or more selected actions within the real-world environment, e.g., to manipulate, move, fixate, and / or reconfigure objects. The augmented reality environment can be displayed to a user, e.g., via a head-mounted or head-up display.

[0090] As yet another example, the environment can be a computing environment, e.g., one or more computing devices optionally connected by a wired or wireless network, and the agent can be a software agent that executes within the computing environment to interact with a user. For example, the agent can be digital assistant software that performs user-specified tasks within the computing environment by performing actions that control one or more of the computing devices.

[0091] 2 is a flow diagram of an exemplary process 200 for selecting one or more actions at a time step. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, an action selection system, such as action selection system 100 of FIG. 1, suitably programmed in accordance with this specification, can perform process 200.

[0092] The system can execute the process 200 at each of multiple time steps to control the agent.

[0093] The system receives an observed image that characterizes the state of the environment at a time step (step 202).

[0094] The system receives a natural language text sequence for a time step that characterizes a task to be performed by an agent in an environment at the time step (step 204).

[0095] The system then processes the observed image and the policy input, including the natural language text sequence, using a policy neural network to select one or more actions to be performed by the agent in response to the observed image.

[0096] An exemplary technique for selecting one or more actions is now described.

[0097] The system processes the observed image and natural language text sequence using a perceptual encoder neural network to generate encoded representations during the time steps (step 206).

[0098] In some implementations, the perceptual encoder neural network includes an image embedding neural network that generates image embeddings that represent the image observations, a text embedding neural network that generates text embeddings that represent the image observations, and a multimodal Transformer neural network that generates aggregate embeddings (which serve as the encoded representations).

[0099] As used herein, an "embedding" is a vector of numerical values, e.g., floating-point or other values, with a given dimensionality. The space of possible vectors with a given dimensionality is referred to as the "embedding space." Image and text embeddings generated by image and text embedding neural networks are typically in the same embedding space.

[0100] The generation of the encoded representation when the perceptual encoder neural network has the above architecture will be described below with reference to FIG.

[0101] The system uses the aggregate embedding to select one or more actions to be performed by the agent in response to the observed image (step 206), and causes the agent to perform the one or more selected actions (step 208).

[0102] In general, the system processes a state representation derived from an encoded representation using a policy neural network to select one or more actions.

[0103] That is, the system generates a state representation from the encoded representation and uses this state representation to select one or more actions by processing the state representation using, for example, a policy neural network. In some embodiments, the state representation is the same as the encoded representation. In some other embodiments, the system generates the state representation by processing the encoded representation using a memory neural network, e.g., a neural network that allows the state representation to incorporate information from previous time steps. For example, the memory neural network can be a recurrent neural network, e.g., a long-short-term memory (LSTM) neural network or a gated recurrent unit (GRU) neural network, to allow the state representation to incorporate information from previous time steps.

[0104] In some implementations, the system selects only a single action at each time step. In these implementations, the policy neural network can be, for example, a multilayer perceptron (MLP) or other feed-forward neural network that generates a probability distribution over the set of actions, and the system can use the probability distribution to greedily select or sample an action. An alternative architecture for the policy neural network when a single action is selected is described below with reference to FIG. 4.

[0105] In some other implementations, the system selects a sequence of multiple actions at each time step. That is, the system generates a sequence of multiple actions that includes respective actions at multiple locations and has the agent perform the sequence of actions in sequential order, e.g., by first performing the action at the first location, then the action at the second location, and so on. An example architecture for a policy neural network capable of generating a sequence of multiple actions is described below with reference to FIG. 4.

[0106] Although this specification generally describes observations as images, in some cases, the observations may include additional data in addition to image data, such as proprioceptive data characterizing the agent or other data captured by other sensors of the agent. In these cases, the other data may be jointly embedded with the observed image by an image embedding neural network.

[0107] 3 is a flow diagram of an exemplary process 300 for generating encoded representations at time steps using a perceptual encoder neural network. For convenience, process 300 is described as being performed by one or more computer systems located at one or more locations. For example, an action selection system, such as action selection system 100 of FIG. 1, suitably programmed in accordance with this specification, can perform process 300.

[0108] In the example of Figure 3, the perceptual encoder neural network includes an image embedding neural network that generates image embeddings that represent image observations, a text embedding neural network that generates text embeddings that represent image observations, and a multimodal Transformer neural network that generates aggregate embeddings (which serve as encoded representations).

[0109] More specifically, the system processes the observed image using an image embedding neural network to generate multiple image embeddings that represent the observed image (step 302). For example, the image embedding neural network can be a convolutional neural network, such as a ResNet or Inception neural network, that processes the observed image to generate a feature map that includes a respective image embedding for each of multiple regions in the observed image. As another example, the image embedding neural network can be a Vision Transformer neural network that processes a sequence of patches from the observed image to generate a respective image embedding for each patch in the sequence.

[0110] The system processes the natural language text sequence using a text embedding neural network to generate multiple text embeddings that represent the natural language text sequence (step 304). For example, the text embedding neural network can use a trained embedding table to map text tokens, e.g., words or word pieces, in the text sequence to embeddings. As another example, the text embedding neural network can be an encoder-only Transformer that processes the sequence of text tokens to generate a respective text embedding for each text token.

[0111] The system processes the input, including image embeddings and text embeddings, using a multimodal Transformer neural network to generate aggregate embeddings that serve as the encoded representation (step 306).

[0112] Transformer neural networks are called "multimodal" because they receive as input embeddings of inputs of multiple different modalities, for example, embeddings of both natural language text and images.

[0113] In some other implementations, the perceptual encoder neural network may use a different type of multimodal neural network, for example a convolutional neural network or a recurrent neural network, to generate aggregate embeddings from at least the image and text embeddings.

[0114] More specifically, the multimodal Transformer neural network is configured to (i) apply self-attention across at least the text embeddings and the image embeddings to generate updated embeddings for at least each of the plurality of text embeddings, and (ii) generate an aggregate embedding from the updated embeddings for at least each of the text embeddings.

[0115] In particular, since the multimodal Transformer includes one or more self-attention layers, each having one or more self-attention heads, "applying self-attention" includes processing an input through one or more self-attention layers. That is, each self-attention layer can perform single-head self-attention, thus having only one attention head, or can perform multi-head attention, thus having multiple heads each performing self-attention in parallel. The self-attention layer can then combine the outputs of the multiple heads to generate the output of the attention mechanism of the self-attention layer, for example, by summing, averaging, or concatenating the outputs and then optionally applying a linear transformation to the result. Each self-attention layer can also perform any of a variety of other operations, such as layer normalization, position-wise feedforward neural network calculations, and residual connection operations.

[0116] Each head of each self-attention layer can apply any of a variety of self-attention mechanisms to at least the inputs corresponding to image embeddings and text embeddings. We now describe one example of such an attention mechanism.

[0117] In this example, the inputs to the multimodal Transformer neural network include image embeddings, text embeddings, and one or more dedicated embeddings. A "dedicated" embedding is one that is the same at each time step and does not depend on the observations at that time step. For example, the dedicated embedding(s) can be learned during training of the neural network or can be fixed to predetermined values.

[0118] The multimodal transformer then applies self-attention to generate updated embeddings for each of the text embedding and the private embedding, without updating the image embedding.

[0119] As an example of this, each self-attention head of each self-attention layer can be configured to receive head inputs including (i) image embeddings generated by the image embedding neural network and (ii) current embeddings for the text embeddings and dedicated embeddings, respectively. That is, even if the head is not present in the first self-attention layer, the head still receives the original image embeddings, but also the current embeddings for the text embeddings and dedicated embeddings updated by the previous self-attention layer(s).

[0120] The head then generates a respective query from each current embedding, corresponding to each text embedding and each dedicated embedding, for example, by applying the learned query linear transformation to each current embedding.

[0121] The head also generates a respective key corresponding to each image embedding, each text embedding, and each dedicated embedding from the image embedding and each current embedding, for example by applying a learned key linear transformation to each embedding.

[0122] The head also generates respective values ​​corresponding to each image embedding, each text embedding, and each dedicated embedding from the image embedding and each current embedding, for example by applying a learned value linear transformation to each embedding.

[0123] The head then applies query-key-value attention to each query, key, and value to generate respective initial updated embeddings for each text embedding and each dedicated embedding, without updating the image embeddings. That is, the head applies self-attention to the text and dedicated embeddings, but only applies "cross-attend" to the image embeddings (because the image embeddings are only used to generate keys and values, not queries).

[0124] With multiple heads, the self-attention layer can combine each initial update embedding as above.

[0125] Self-attention and query-key-value attention are described in more detail below.

[0126] Once the multimodal Transformer generates updated embeddings for each of the text embeddings and the private embeddings, the Transformer can generate aggregate embeddings from each of the updated embeddings.

[0127] As a particular example, the Transformer can aggregate the updated embeddings of each of the text embeddings and the private embeddings to generate an initial aggregate embedding, and then combine the updated embeddings of each of the private embeddings with the initial aggregate embedding to generate an aggregate embedding.

[0128] The Transformer can apply any of a variety of aggregation operations, such as a pooling operation, to each of the updated text embeddings and the dedicated embeddings to generate the initial aggregated embeddings. For example, the Transformer can apply feature-wise average pooling to each of the updated text embeddings and the dedicated embeddings to generate the initial aggregated embeddings.

[0129] The Transformer can combine each updated embedding of the private embedding with the initial aggregate embedding in any of a variety of ways to generate an aggregate embedding. As an example, the Transformer can concatenate each updated embedding with each of the private embedding and the initial aggregate embedding.

[0130] The system then uses the aggregate embedding as the state representation.

[0131] 4 is a flow diagram of an exemplary process 400 for generating a sequence of multiple actions at a time step. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, an action selection system (e.g., action selection system 100 of FIG. 1 ) can be suitably programmed in accordance with this specification to perform process 400.

[0132] The system receives an observed image for a time step that characterizes the state of the environment at the time step (step 402).

[0133] The system receives a natural language text sequence for a time step that characterizes a task to be performed by an agent in an environment at the time step (step 404).

[0134] The system processes the observed images and natural language text sequences to generate state representations during the time steps (step 406). For example, the system can generate the state representations as described above with reference to FIG. 2, or using a different set of neural networks with different architectures.

[0135] The system generates a sequence of actions to be performed by the agent in response to the observed image at a time step. As described above, the sequence has a respective action taken by the agent at each of a plurality of locations. In the example of Figure 4, the policy neural network includes a high-level controller neural network and an action-policy neural network to implement a hierarchical action selection scheme, e.g., a hierarchical control scheme.

[0136] In particular, the system processes the state representation using a high-level controller neural network to generate respective low-level inputs for each position in the sequence (step 408). As a particular example, the high-level controller neural network can receive the state representation as input and then autoregressively generate respective low-level inputs for each position in the sequence. "Autoregressively" generating low-level inputs refers to generating inputs for each position that are conditioned on the inputs of all positions preceding that position in the sequence. For example, the high-level controller neural network can be a recurrent neural network, such as an LSTM or GRU, that receives the state representation as input at a first processing time step and, at each subsequent processing time step, receives as input the low-level inputs generated at the previous processing time step.

[0137] For each position, the system processes the position's respective low-level inputs using an action policy neural network to generate an action to be performed by the agent at that position in the sequence (step 410). In some implementations, each action is composed of multiple sub-actions. For example, if the agent is a robot or other mechanical agent, the sub-actions may include two or more of a grab action to attempt to grasp an object in the environment, a push / pull action to push or pull an object into the environment, a rotate action to rotate one or more parts of the agent's body, a look action to change the orientation of the agent's camera, and a move action to move the agent within the environment. The agent's control system can map these high-level actions to low-level commands, such as torques on the agent's joints or other forces applied to parts of the agent's body, to control the agent.

[0138] In these implementations, the action policy neural network can include a respective subnetwork for each of the plurality of sub-actions. Thus, for each of the plurality of sub-actions, to process the respective low-level inputs for the position using the action policy neural network, the action policy neural network processes inputs including the respective low-level inputs for the position using the sub-action subnetwork to select a value for the sub-action for the position. For example, each subnetwork can be configured to generate an output that defines a probability distribution over the possible values ​​of the corresponding sub-action, and the system can use the probability distribution to greedily select values ​​for the sub-action or sample the sub-action.

[0139] In some of these implementations, for at least one of the subactions, the input includes a selected value for one or more of the other subactions at that location, i.e., the value of at least one subaction at a given location can depend on the value of at least one other subaction at the given location.

[0140] When only a single action is selected per time step, the policy neural network can either (i) include only a respective sub-network for each of multiple sub-actions, and the input for each sub-network can include a state representation (instead of a low-level input), or (i) include a feedforward high-level controller that maps the state representation to a single low-level input at a single location.

[0141] 5 is a flow diagram of an exemplary process 500 for training a policy neural network with the output of a reward neural network using reinforcement learning. For convenience, process 500 is described as being performed by one or more computer systems located at one or more locations. For example, a training system, such as training system 190 of FIG. 1, suitably programmed in accordance with this specification can perform process 500.

[0142] In some implementations, before training the neural network by reinforcement learning, the system trains the policy neural network by imitation learning, for example, on ground truth data generated by an expert agent.

[0143] Thus, in these embodiments, the system acquires ground truth data, which includes a set of ground truth trajectories, each including observations having observed images and natural language text sequences, at a sequence of time steps, and one or more of ground truth actions or ground truth text outputs (step 502). A "ground truth" action is a target action that needs to be performed by an agent at a given time step (or a given position in an action sequence). Similarly, a "ground truth" text output is a target text output that needs to be generated by the system at a given time step. For example, the ground truth action and text output can be the actual action and text output performed or generated (e.g., spoken) (respectively) by an expert agent at a given time step. The expert agent can be, for example, an agent controlled by a human user, an agent controlled by a learned policy, or an agent controlled by a hard-coded, heuristic-based policy.

[0144] The system can then train the policy neural network by imitation learning based on the ground truth data (step 504). For example, the system can use "supervised learning" to score the ground truth actions and ground truth text output using each policy. That is, the system can train the policy neural network using a behavior cloning loss or other suitable imitation learning loss. The imitation learning loss can include, for example, one or more terms that compare the actions and / or text output generated by the system at a given time step to the ground truth actions and / or text output at that time step.

[0145] The system trains the policy neural network by reinforcement learning using the reward values ​​generated by the reward neural network, step 506. An exemplary process for training the reward neural network to accurately estimate rewards is described in detail with reference to Figures 6 and 7.

[0146] For example, when controlling an agent during reinforcement learning training, the system can generate a per-time-step reward for each given time step from at least the reward values ​​for the time steps generated by the reward neural network, and then train a policy neural network using the per-time-step rewards using any suitable reinforcement learning algorithm, i.e., by using the per-time-step rewards as rewards provided as input to the reinforcement learning algorithm.

[0147] For example, for each of a plurality of time steps, the system may calculate a reward for each time step based on the difference between the reward value at that time step and the reward value of the preceding time step in the sequence, e.g., the immediately preceding time step or the time step a specified number of time steps before the time step in the sequence.

[0148] The reinforcement learning algorithm can be any suitable reinforcement learning algorithm, for example, an actor-critic RL algorithm, a policy gradient-based RL algorithm, etc.

[0149] Training the policy neural network by reinforcement learning using a reward neural network is described in more detail below with reference to FIG. 8.

[0150] 6 is a flow diagram of an exemplary process 600 for obtaining training data for training a reward neural network. For convenience, process 600 is described as being performed by one or more computer systems located at one or more locations. For example, a training system, such as training system 190 of FIG. 1, suitably programmed in accordance with this specification can perform process 600.

[0151] The system acquires interaction data between agents in an environment at a sequence of time steps during multiple training task episodes (step 602). In some embodiments, within each episode, one agent sets a task (setter) for a second agent (solver) to complete. In some embodiments, the task can be free-form, allowing the setter to select a task or ask questions at will. In other embodiments, the task can prompt the system to provide a high-level director for the setter to follow.

[0152] For example, the system may prompt the setter to provide the solver with the following command: "Ask another agent to hand you something."

[0153] In some implementations, the setter is free to give new instructions to the solver at will in response to the solver's actions.

[0154] In some implementations, the setter agent may be controlled by a setter neural network. The setter training system may train the setter neural network using imitation learning based on the actions of an expert setter.

[0155] Generally, an agent is referred to as an "interactive" agent because, as part of interacting with the environment, the agent interacts with one or more other agents in the environment. The one or more other agents may include humans, other agents controlled by different computer systems, or both. An interactive agent interacts with other agent(s) by receiving communications generated by the other agent(s) and, optionally, generating text that is communicated to the other agent(s). In some implementations, the agent(s) may be controlled by a human user or by an expert policy (i.e., a fixed, rule-based policy or a previously learned policy).

[0156] The system obtains annotation data from one or more evaluators (step 604). The one or more evaluators can be users who observe interactions between the setter and the solver, and the annotation data can include annotations that evaluate the progress of the solver against directives at one or more time steps within the training task episode.

[0157] For example, the system may instruct the setter to provide the command "Ask the other agent to hand you something." In this scenario, during one or more time intervals within a training task episode (the period after the setter delivers the command to the solver and before the task is either completed or interrupted), the evaluator evaluates whether the solver performs actions that help it progress toward the instructed goal.

[0158] In general, the annotation data for a given time step assigns a reward label to the time step.

[0159] More specifically, the annotation data assigns, to each of a plurality of time steps in the task episode, a respective reward label selected from a set of reward labels including: (i) a negative reward label indicating that, at the time of the time step, the agent has regressed from achieving the goal characterized in the natural language instructions for the time step, and (ii) a positive reward label indicating, at the time of the time step, the agent has made progress in achieving the goal characterized in the natural language instructions for the time step. Optionally, the set also includes (iii) a neutral reward label indicating that the agent has made neither progress nor regress from achieving the goal.

[0160] The system uses the training data (annotated agent-agent interactions from one or more evaluators at multiple time steps within a training task episode) to train a reward neural network (step 606).

[0161] 7 is a flow diagram of an exemplary process 700 for training a reward neural network. For convenience, process 700 is described as being performed by one or more computer systems located at one or more locations. A training system, such as, for example, training system 190 of FIG. 1, suitably programmed in accordance with this specification, can perform process 700.

[0162] The system can repeatedly run process 700 on different mini-batches of training data to train a reward neural network that is used to train a policy neural network via reinforcement learning.

[0163] The system trains a reward neural network to determine trained values ​​for a set of parameters of the neural network, e.g., weights and biases for layers of the neural network. In some implementations, the system initializes the parameters of the reward neural network using parameter values ​​of the policy neural network determined by training the policy neural network through imitation learning.

[0164] For a given training task episode, the system receives a sample of observations and corresponding annotations from the evaluators from the task episode (step 702). In particular, the sample of observations includes a sequence of observations, each sequence of observations including a respective observation for each of a plurality of time steps in the training task episode. Thus, for each time step, the system receives multimodal input, e.g., input including data of multiple modalities (e.g., data types), including at least an observation image characterizing the state of the environment at the time step and a natural language text sequence. Each annotation is generally associated with a respective time step (and thus with an observation at the time step).

[0165] For example, a training task episode may include a setter commanding a solver to "pick up a tissue roll and place it on the bathtub." The evaluator observes interactions between the setter and solver at multiple points within the training task episode and marks key moments when the solver moves forward toward the presented command and marks key moments when the solver moves backward away from the presented command. If the solver drops the tissue roll on the floor, the evaluator may mark this action as a backward move away from the presented command. However, if the solver picks up the tissue roll from the floor and begins walking toward the bathtub, the evaluator may mark this action as a forward move toward the presented command.

[0166] As described above, the system configures the reward neural network to output reward values ​​that direct the solver's progress toward the presented instructions at specific times within the training task episode.

[0167] In particular, reward values ​​can measure progress (also called "utility") across a time-step trajectory; i.e., the reward value for a given time step represents the utility of performing the task of the trajectory of observations up to the observation for that time step. For example, the utility of a trajectory beginning at the start of a training task episode and ending at some time "t1" indicates how much progress was made toward the presented instructions within that time interval, i.e., the time interval from the beginning of the training episode start to time t1.

[0168] The system defines a loss function that depends on the reward labels during the time steps within each of the multiple trajectories in the episode (step 704). The loss function can be written as: L(θ)=E D [prefer(x ≦t ,x ≦t’ )ln(σ(U θ (x≦t )-U θ (x x≦t’ ))] In the formula, x ≦t =(x0,...,x t ) is a sequence of observations (images and dialogue) up to time t, where t and t' are two time points on the same trajectory, D is the dataset of rater annotations, and prefer(x ≦t ,x ≦t’ ) is an indicator function that is 1 if t' follows t and is marked as positive, or -1 if t follows t' and is marked as negative.

[0169] For example, the loss function can measure the difference between the reward values ​​predicted for two time steps within the same training task episode.

[0170] In this example, if both time steps in a pair have positive reward labels and no time steps between the time steps in the pair have negative reward labels, the loss function encourages the predicted reward value for the later time step in the pair to be greater than the predicted reward value for the earlier time step in the pair. Furthermore, if both time steps in a pair have negative reward labels and no time steps between the time steps in the pair have positive reward labels, the loss function encourages the predicted reward value for the earlier time step in the pair to be greater than the predicted reward value for the later time step in the pair. In some implementations, any pairs of time steps that do not meet either criterion can be ignored. In some other implementations, if both time steps in a pair have neutral reward labels and all time steps between the time steps in the pair have neutral reward labels, the loss function encourages the predicted reward value for the later time step in the pair to be equal to the predicted reward value for the earlier time step in the pair.

[0171] Rather than measuring the difference between pairs of time steps, the loss function can instead measure, for each time step within each of the training episodes, the error between the reward label for that time step and the reward value for that time step.

[0172] In some implementations, the system uses one or more auxiliary losses when training the reward neural network. For example, the auxiliary losses can include an imitation learning loss calculated using outputs generated by an auxiliary policy neural network head that generates policy outputs for controlling the agent from intermediate outputs of the reward neural network.

[0173] As another example, the auxiliary loss can include a contrast self-supervised representation learning loss, whose formula can be written as follows:

number

[0174] Similarly, the system may define auxiliary losses to include a cross-modality matching loss that uses the output generated by the reward neural network.

[0175] The system trains a reward neural network based on the loss function (step 706). To train the reward neural network, the system calculates the gradient of the loss function by evaluating the loss function at different time steps within the training episode. The system then updates the parameters of the reward neural network by applying an optimizer, such as Adam, AdamW, or Adafactor, to the gradient.

[0176] 8 is a flow diagram of an exemplary process 800 for training a policy neural network by reinforcement learning using reward values ​​generated from a reward neural network. For convenience, process 800 is described as being performed by one or more computer systems located at one or more locations. For example, a training system (e.g., action selection system 100 of FIG. 1 ) can be suitably programmed in accordance with this specification to perform process 800.

[0177] The system receives an observed image for a time step that characterizes the state of the environment at the time step (step 802).

[0178] The system receives a natural language text sequence for a time step that characterizes a task to be performed by an agent in an environment at the time step (step 804).

[0179] The system processes the policy input, including the observed image and the natural language text sequence, using a policy neural network to generate an output that determines one or more actions to be taken by an agent acting in the environment as described above with reference to Figures 1-3 (step 806).

[0180] The system processes the reward input, which includes the observed image and the natural language text sequence, using a reward neural network (step 808). As described above, the system trains the reward neural network such that the reward neural network is configured to process the observed image and the natural language text to generate a reward output that defines a reward value that characterizes the agent's progress in performing the task characterized by the natural language text sequence at the time step.

[0181] An exemplary architecture of a reward neural network is described in more detail below with reference to FIG.

[0182] The system trains the policy neural network using reinforcement learning techniques (step 810).

[0183] 9 is a flow diagram of an exemplary process 900 for generating reward values ​​using a reward neural network. For convenience, process 900 is described as being performed by one or more computer systems located at one or more locations. For example, a training system, such as reward neural network training system 190 of FIG. 1, suitably programmed in accordance with this specification, can perform process 900.

[0184] The reward neural network can generally have any suitable architecture that enables the reward neural network to map images and text to scalar reward values.

[0185] In particular, in the example of Figure 9, the reward neural network has the same architecture as the policy neural network, but the policy head (sub-neural network) is replaced with a reward head (sub-neural network). In some cases, parameters of components shared between the policy neural network and the reward neural network can be initialized using values ​​of parameters determined by pre-training the policy neural network, for example, through imitation learning.

[0186] The system uses an image embedding neural network to generate multiple image embeddings that represent the observed image (step 902), for example, as described above.

[0187] The system processes an input including a natural language text sequence using a text embedding neural network to generate a plurality of text embeddings that represent at least the natural language text sequence (step 904), e.g., as described above.

[0188] The system processes the input, including image embeddings and text embeddings, using a multimodal neural network to generate aggregate embeddings (step 906).

[0189] The multimodal neural network may generally have any suitable architecture.

[0190] As an example, the multimodal neural network can be a multimodal Transformer neural network configured to (i) apply self-attention across at least the text embeddings and the image embeddings to generate updated embeddings for at least each of the plurality of text embeddings, and (ii) generate an aggregate embedding from the updated embeddings for at least each of the text embeddings.

[0191] As described above, a multimodal Transformer neural network includes one or more self-attention layers, each with one or more self-attention heads, and applying self-attention includes processing an input through the one or more self-attention layers.

[0192] As mentioned above, in some cases, the inputs to the multimodal Transformer neural network include image embeddings, text embeddings, and one or more proprietary embeddings.

[0193] As mentioned above, applying self-attention can include generating updated embeddings for each of the text embedding and the dedicated embedding without updating the image embedding.

[0194] In this example, each self-attention head of each self-attention layer is configured to receive head input including (i) image embeddings generated by the image embedding neural network and (ii) current embeddings for each of the text embeddings and dedicated embeddings; to generate, from the respective current embeddings, respective queries corresponding to each of the text embeddings and dedicated embeddings; to generate, from the image embeddings and the respective current embeddings, respective keys corresponding to each of the image embeddings, each of the text embeddings, and each of the dedicated embeddings; to generate, from the image embeddings and the respective current embeddings, respective values ​​corresponding to each of the image embeddings, each of the text embeddings, and each of the dedicated embeddings; and to apply query-key-value attention across the respective queries, keys, and values ​​to generate respective initial updated embeddings for each of the text embeddings and each of the dedicated embeddings without updating the image embeddings.

[0195] In some examples, to generate the aggregated embeddings, the system aggregates updated embeddings for each of the text embeddings and the dedicated embeddings to generate an initial aggregated embedding, and combines updated embeddings for each of the dedicated embeddings with the initial aggregated embedding to generate the aggregated embeddings.

[0196] In some examples, the system combines each of the updated embeddings and each of the initial aggregate embeddings by concatenating each of the updated embeddings and each of the initial aggregate embeddings.

[0197] Further details regarding the operation of the Multimodal Transformer are described above with reference to the description of policy neural networks.

[0198] The system then processes the input, including the aggregate embedding, using a reward neural network head to generate a reward value (step 908).

[0199] In some examples, the reward neural network head generates a state representation from the aggregate embeddings and processes the state representations using one or more neural network layers to generate a reward output. For example, the head can generate the state representations by processing the aggregate embeddings using a memory neural network.

[0200] As noted above, the memory neural network can be a recurrent neural network.

[0201] Figure 10 shows a comparison of the performance of different versions of the policy neural network.

[0202] Specifically, Figure 10 shows the performance of an agent controlled by a policy neural network trained using imitation learning (denoted as behavioral cloning, BC), imitation learning fine-tuned by reinforcement learning (BC+RL), and a comparison with a human-controlled agent performing the same actions. In particular, the plot in Figure 10 compares an agent controlled by a policy neural network trained solely on imitation learning with an agent controlled by a policy neural network trained using both imitation learning from a reward neural network and reinforcement learning, demonstrating the performance improvement between the agents. Agents controlled by policy neural networks trained with both imitation learning and reinforcement learning more closely resemble human performance compared with agents controlled by policy neural networks trained solely with imitation learning. Agents controlled by policy neural networks trained with imitation learning and reinforcement learning achieved an 89% success rate, while agents controlled by policy neural networks trained solely with imitation learning achieved a 78% success rate. This compares to a 96% success rate for humans solving the same task.

[0203] The agents in this evaluation were evaluated by expert evaluators across a variety of tasks belonging to a class of tasks where the instructions and task completion were clearly defined. For example, common tasks include counting, identifying colors, lifting objects, and positioning one object next to another. Evaluators can easily assess whether the agents are successful in completing these tasks where the objectives are clear.

[0204] 11 illustrates an exemplary process for improving a policy neural network by reinforcement learning using the output of a reward model. For convenience, the process is described as being performed by one or more computer systems located at one or more locations.

[0205] The process can be repeated over multiple training iterations. In each iteration, the system deploys the agent to interact with humans and collects feedback on its behavior from evaluators. The human-agent interaction can be of a set of multiple prompts, including language game prompts. For example, the system can provide the agent with a prompt consisting of a text cue indicating a general command type combined with a modifier that specifies additional constraints that the human command must satisfy.

[0206] The system can employ various language games to form the basis of human-agent interaction, which can include command-following tasks such as: "ask the other player to touch an object with another object," "ask the other player to perform an activity of your choice," "ask the other player to hand you something you were holding at the time," or "ask the other player to stand in a position relative to you." Alternatively or additionally, language games can include question-answer tasks such as: "ask the other player a yes / no question about something in the room," "ask the other player to describe where something is," "ask the other player to count something," "ask the other player to say what they are looking at or paying attention to," or "ask about the color of something." As another type of language game, human-agent interaction can include tasks aimed at modifying the agent's behavior, such as "Show me the object by color," "Show me the location by color," "Show me the object by location," or "Use shape words. Try using shape words such as circle, rectangle, round, pointy, long, etc."

[0207] The system updates the reward neural network by training it based on data collected by human-agent interaction data with feedback from evaluators.

[0208] The policy network can be continuously updated as the reward neural network is updated with more human-agent interaction data. The system can evaluate the output of the trained policy network by reinforcement learning using the reward model output. As more data is collected from human-agent interactions, the reward neural network is updated, and the updated reward neural network is used to retrain the policy neural network.

[0209] FIG. 12 shows a diagram of an exemplary process for training a policy neural network, training a reward neural network, and training the policy neural network with the reward neural network using reinforcement learning.

[0210] The policy neural network is first trained using imitation learning, which is often called behavior cloning, which frames behavior copying as a sequence learning problem. Policy neural networks trained by imitation learning create agents that capture the diversity of human interaction behaviors.

[0211] The system collects human-agent interactions as described with respect to Figure 11. Human-agent interactions between a policy neural network trained by imitation learning and a human agent. The human-agent interactions are evaluated by evaluators as described with respect to Figure 6.

[0212] The reward neural network is then trained on human-agent interaction data annotated by the evaluators as described with respect to FIG. 7.

[0213] The output of the reward neural network is then used to train the policy neural network using reinforcement learning as described with respect to FIG.

[0214] The system began by training an agent via behavioral cloning (BC) from a dataset of human-human interactions. It then collected a dataset of human-agent interactions and, for a subset of episodes, asked humans to provide judgments of progress toward or retreat from a human-commanded goal. It then trained a network to model this human feedback and obtained a reward model. Finally, it used the reward model to train a new agent that combined behavioral cloning and reinforcement learning from human feedback (BC+RL).

[0215] Briefly, we trained an imitation-based agent via behavioral cloning (BC) from a dataset of human-human interactions, following a similarly cited method. In these human-human interactions, one player (the setter) set tasks for a second player (the solver), who then performed the tasks. These tasks included mobile manipulation and question-answering, or a combination of these. Imitation learning often produced agents competent in human interactions. We then asked humans to interact with this imitation-trained agent. Human evaluators annotated a subset of these interactions offline by viewing videos from the solver agent's perspective. The evaluators marked discrete moments when the agent advanced toward or retreated from a goal. We modeled these annotated data to obtain a "reward model" that captured the details of human feedback. Finally, we used reinforcement learning to train the agent to improve with respect to the output of the learned reward model.

[0216] Figure 13 shows an example of data collected to train the reward neural network. It shows an example episode in the rater feedback dataset. The original interaction between the human and the agent is shown to the left of a plot of the feedback given to a particular episode by six human raters. Positive and negative feedback marks are represented by solid green and empty red dots, respectively. Although six ratings were collected for an episode, most of the data was annotated by a single rater (1.2 raters on average).

[0217] The figure shows an example of an episode annotated by several evaluators. As shown in the zoomed-in segment, when the human setter asks the agent solver to "pick up the tissue roll and place it on the bathtub," the evaluators marked as positive key moments the agent came closer to solving the task, such as picking up the object and placing it in the correct place. They also marked as negative key moments any setbacks, such as dropping the object on the floor. In addition to collecting feedback on human-agent interactions, we also solicited human feedback on human-human and human-agent-human interactions. Human-human interactions typically indicate long sequences of desired behavior, while human-agent-human interactions typically indicate mistakes in agent behavior and the associated correction of those mistakes. In total, we collected 5,104,000 individual feedback marks across 364,690 episodes.

[0218] Figure 14 shows an example pseudocode algorithm for implementing the "Inter-temporal Bradley-Terry" model to train a reward neural network. The loss function at each training step is defined by human rater annotations of interactions, which depend on whether the human rater rated the interaction as positive or negative for progress toward the stated goal. The system can use reinforcement learning to optimize the relative change in utility during the episode, U(xT) - U(x0). Here, the reward per step is given by r = U(xt+1) - U(xt), where the utility U(xt) at each time step is provided by the IBT reward model.

[0219] The term "configured" is used herein in connection with systems and computer program components. When one or more computer systems are configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when running, causes the system to perform the operation or action. When one or more computer programs are configured to perform a particular operation or action, it means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0220] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, such as the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more modules of computer program instructions, i.e., as one or more computer programs encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial-access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, such as a mechanically generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a receiving device suitable for execution by the data processing apparatus.

[0221] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or further include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus optionally includes code that creates an execution environment for a computer program (e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof).

[0222] A computer program, which may be called or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, for example, files that store one or more modules, subprograms, or code portions. A computer program can be deployed to be executed on one computer or to be executed on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communications network.

[0223] As used herein, the term "database" is used broadly to refer to any collection of data. The data need not be structured in any particular way, or even at all, and may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0224] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and run on the same computer or computers.

[0225] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA or an ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0226] A computer suitable for running a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or on other types of central processing units. Generally, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The basic components of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Furthermore, a computer may be incorporated in other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0227] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0228] To provide for interaction with a user, embodiments of the subject matter described herein may be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to interact with the user. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer may interact with a user by sending documents to a device used by the user and receiving documents from a device used by the user (e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser). A computer may also interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and then receiving a reply message from the user.

[0229] A data processing device for implementing machine learning models may also include dedicated hardware accelerator units, for example for handling the common and computationally intensive part of machine learning training or production, i.e., inference, workloads.

[0230] The machine learning model can be implemented and deployed using a machine learning framework, for example, TensorFlow or the JAX framework.

[0231] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an embodiment of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), e.g., the Internet.

[0232] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to a user device, for example, to display the data to and receive user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., results of user interaction, can be received from the device by the server.

[0233] While the specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of patentable subject matter, but rather as descriptions of features that may be unique to particular embodiments of a particular invention. Certain features described herein as separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented in multiple embodiments, either separately or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of the subcombinations.

[0234] Similarly, while operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0235] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still produce desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method, executed by one or more computers, for training a policy neural network for controlling an agent interacting with an environment, comprising: At each of multiple time steps during a task episode, obtaining an observation for the time step, the observation comprising: an image characterizing a state of the environment at said time step; a natural language text sequence for the time step that characterizes a task being performed by the agent in the environment at the time step; processing a policy input comprising the observed image and the natural language text sequence using the policy neural network to select one or more actions to be performed by the agent in response to the observed image; processing a reward input comprising the observed image and the natural language text sequence using a reward neural network, the reward neural network being configured to process the observed image and the natural language text sequence to generate a reward output defining a reward value that characterizes progress of the agent in performing the task characterized by the natural language text sequence at the time step; training the policy neural network by reinforcement learning using the respective reward values ​​for the plurality of time steps.

2. 10. The method of claim 1, further comprising, prior to training the policy neural network by reinforcement learning, training the policy neural network by imitation learning on data characterizing interactions between multiple agents in the environment.

3. 3. The method of claim 1, wherein the reward neural network is configured to generate a reward value that represents a utility for performing the task of a trajectory of observations up to the observation for the time step.

4. training the policy neural network by the reinforcement learning using the respective reward values ​​for the plurality of time steps; for each of the plurality of time steps, calculating a per-time-step reward based on a difference between the reward value at that time step and the reward value for a preceding time step in the sequence; training the policy neural network by reinforcement learning using the respective time-step rewards for the plurality of time steps; The method according to any one of claims 1 to 3, comprising:

5. The method of any one of claims 1 to 4, wherein the reward input further comprises a reward value generated for a previous time step in the sequence.

6. The method of any one of claims 1 to 5, wherein the reward input further comprises a natural language output produced by the agent in a previous time step.

7. The method of any one of claims 1 to 6, wherein the reward output is the reward value.

8. A method according to any preceding claim, wherein the reward output comprises a probability distribution over a set of possible reward values.

9. The reward neural network, for each of one or more time steps within each of a plurality of training task episodes, (i) a negative reward label indicating that, at the time step, the agent has regressed from achieving a goal characterized in the natural language instruction for that time step; and (ii) a positive reward label indicating, at the time step, that the agent has made progress in achieving the goal characterized in the natural language instruction for that time step; and The method of any one of claims 1 to 8, wherein the method is trained with training data comprising respective reward labels selected from a set of reward labels comprising:

10. The set also includes: (iii) a neutral reward label; 10. The method of claim 9, comprising:

11. 11. The method of claim 9, wherein the reward model is trained on the training data with a loss function that measures the difference between predicted reward values ​​for two time steps within the same training task episode.

12. For a given pair of time steps within the same training task episode, If both time steps in the pair have positive reward labels and no intervening time steps in the pair have negative reward labels, the loss function encourages the reward value predicted for a later time step in the pair to be larger than the reward value predicted for an earlier time step in the pair; 12. The method of claim 11 , wherein if both time steps in the pair have negative reward labels and no time steps existing between the time steps in the pair have positive reward labels, the loss function encourages the reward value predicted for the earlier time step in the pair to be larger than the reward value predicted for the later time step in the pair.

13. 13. The method of claim 12, wherein, for the given pair of time steps, if both time steps in the pair have a neutral reward label and all time steps existing between the time steps in the pair have a neutral reward label, the loss function encourages the reward value predicted for the later time step in the pair to be equal to the reward value predicted for the earlier time step in the pair.

14. 11. The method of claim 9, wherein the loss function measures, for each time step within each of the training episodes, the error between the reward label for that time step and the reward value for that time step.

15. processing the observed image and the reward input comprising the natural language text sequence using a reward neural network, using an image embedding neural network to generate a plurality of image embeddings representing the observed images; processing an input including the natural language text sequence using a text embedding neural network to generate a plurality of text embeddings representing at least the natural language text sequence; processing inputs including the image embeddings and the text embeddings using a multimodal neural network to generate aggregate embeddings; processing an input including the aggregate embedding using a reward neural network head to generate the reward value; The method according to any one of claims 1 to 14, comprising:

16. 16. The method of claim 15, wherein the multimodal neural network is a multimodal Transformer neural network, the multimodal Transformer neural network configured to: (i) apply self-attention across at least the text embeddings and the image embeddings to generate updated embeddings for at least each of the plurality of text embeddings; and (ii) generate the aggregate embedding from the updated embeddings for at least each of the text embeddings.

17. 17. The method of claim 16, wherein the multimodal transformer neural network includes one or more self-attention layers each having one or more self-attention heads, and wherein applying the self-attention includes processing the input through the one or more self-attention layers.

18. 18. The method of claim 16 or claim 17, wherein the inputs to the multimodal transformer neural network include the image embeddings, the text embeddings, and one or more proprietary embeddings.

19. 20. The method of claim 18, wherein applying the self-attention comprises generating updated embeddings for each of the text embedding and the dedicated embedding without updating the image embedding.

20. Each self-attention head in each self-attention layer is receiving head input including (i) the image embeddings generated by the image embedding neural network, and (ii) a current embedding for each of the text embeddings and the dedicated embeddings; generating respective queries corresponding to each text embedding and each dedicated embedding from the respective current embeddings; generating a respective key corresponding to each image embedding, each text embedding, and each dedicated embedding from the image embedding and each current embedding; generating respective values ​​corresponding to each image embedding, each text embedding, and each dedicated embedding from the image embeddings and the respective current embeddings; applying query-key-value attention across the respective queries, keys, and values ​​to generate respective initial updated embeddings for each text embedding and each dedicated embedding, without updating the image embeddings; 20. The method of claim 18 or 19, configured to:

21. generating the aggregated embedding aggregating the updated embeddings of the text embedding and the private embedding to generate an initial aggregated embedding; combining the updated embeddings of each of the private embeddings with the initial aggregate embedding to generate the aggregate embedding; 21. The method of claim 20, comprising:

22. 22. The method of claim 21, wherein the combining comprises concatenating each of the updated embeddings with each of the initial aggregate embeddings.

23. processing an input including the aggregate embedding using a reward neural network head to generate the reward output; generating a state representation from the aggregate embedding; processing the state representation using one or more neural network layers to generate the reward output; The method according to any one of claims 15 to 22, comprising:

24. 24. The method of claim 23, wherein generating the state representation includes processing the aggregate embeddings using a memory neural network.

25. 24. The method of claim 23, wherein the memory neural network is a recurrent neural network.

26. 26. The method of any one of claims 15 to 25, wherein the reward input further comprises a natural language output produced by the agent in a previous time step, the input to the text embedding neural network further comprises the natural language output, and the text embeddings represent the natural language text sequence and the natural language output.

27. 27. The method of any one of claims 1 to 26, wherein the natural language text sequences are generated from natural language text sequences that are generated based on corresponding natural language text sequences from corresponding time steps within a training task episode.

28. The method of any one of claims 1 to 27, wherein the natural language text sequences are generated by a setter agent in the environment.

29. 30. The method of claim 28, wherein the setter agent is controlled using a setter neural network.

30. 30. The method of claim 28, wherein the setter neural network is trained to mimic an expert setter agent by imitation learning.

31. 31. The method of any one of claims 1 to 30 when dependent on any one of claims 9 to 14, wherein the reward neural network is trained over a loss function comprising: (i) a loss function that is based on the reward labels; and (ii) one or more auxiliary losses.

32. 32. The method of claim 31 , wherein the one or more auxiliary losses comprise an imitation learning loss calculated using outputs generated by an auxiliary policy neural network head that generates policy outputs for controlling the agent.

33. 33. The method of claim 31 or claim 32, wherein the one or more auxiliary losses comprise a control self-supervised representation learning loss.

34. 34. The method of claim 33 when dependent on claim 15, wherein the one or more auxiliary losses include a cross-modality matching loss that uses outputs produced by the multi-modal neural network.

35. 35. The method of any one of claims 2 to 34 when dependent on claim 2, wherein at least some of the parameter values ​​of the reward neural network are initialized using parameter values ​​of the policy neural network determined by training the policy neural network by imitation learning.

36. training the policy neural network by reinforcement learning using the respective reward values ​​for the plurality of time steps; 36. The method of any one of claims 1 to 35, comprising training the policy neural network by reinforcement learning and by imitation learning on an imitation learning dataset.

37. A method according to any preceding claim, wherein each observed image is captured by a camera sensor on the agent or a camera sensor located within the environment.

38. 38. The method of any one of claims 1 to 37, wherein the agent is a machine agent that interacts with a real-world environment to achieve a specified goal by performing actions selected by the trained policy neural network in response to observations of the real-world environment.

39. 37. The method of any one of claims 1 to 36, wherein the agent is a software agent configured to control electromechanical devices in the real-world environment by performing actions selected by the trained policy neural network in response to observations of the real-world environment.

40. 37. The method of any one of claims 1 to 36, wherein the environment is a computing environment and the agent is a software agent executing within the computing environment to control one or more computing devices by performing actions selected by the trained policy neural network to perform a task specified by a user interacting with the software agent.

41. 40. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of a respective method according to any one of claims 1 to 39.

42. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method operations of any one of claims 1 to 39.

Images