Selecting actions using multi-modal input

Abstract

A method performed by one or more computers for selecting an action to be performed by an agent interacting with an environment, the method comprising, at each of a plurality of time steps: receiving a current string of natural language text expressing something about a current task being performed by the agent; receiving a current observation characterizing a current state of the environment; processing an input comprising the current string of natural language text and the current observation using a policy neural network to generate an action selection output; and selecting an action to be performed by the agent at the time step based on the action selection output; wherein the policy neural network has been trained end-to-end using reinforcement learning.

Description

[0001] This application is a divisional application of the invention patent application filed on June 5, 2018, with application number 201880026852.4 and title "Using Multimodal Input Selection Action". Background Technology

[0002] This specification relates to reinforcement learning. In a reinforcement learning system, an agent interacts with the environment by performing actions selected by the reinforcement learning system in response to observations that represent the current state of the environment.

[0003] Some reinforcement learning systems select the action an agent should perform in response to a given observation, based on the output of the neural network.

[0004] A neural network is a machine learning model that uses one or more non-linear units to predict the output of a received input. Some neural networks are deep neural networks, which include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0005] This specification describes a system implemented as a computer program on one or more computers at one or more locations that selects actions to be performed by a reinforcement learning agent that interacts with the environment.

[0006] According to a first aspect, a system is provided for selecting actions to be performed by an intelligent agent interacting with an environment. The system includes one or more computers and one or more storage devices storing instructions. When the instructions are executed by the one or more computers, the one or more computers implement a language encoder model, an observation encoder neural network, and a subsystem.

[0007] The language encoder model is configured to receive an input text string in a specific natural language and process the input text string to generate a text embedding for the input text string. The observation encoder neural network is configured to receive an input observation representing the state of the environment and process the input observation to generate an observation embedding for the input observation. A subsystem is configured to receive a current text string in a specific natural language, which expresses information about the current task the agent is currently performing. This subsystem provides the current text string as input to the language encoder model to obtain a current text embedding for the current text string. The subsystem receives a current observation representing the current state of the environment. This subsystem provides the current observation as input to the observation encoder neural network to obtain a current observation embedding for the current observation. This subsystem combines the current observation embedding and the current text embedding to generate a current combined embedding. Using the current combined embedding, the subsystem selects the action the agent should perform in response to the current observation.

[0008] In some embodiments, the instructions also cause the one or more computers to implement an action selection neural network. The action selection neural network is configured to receive input combination embeddings and process the input combination embeddings to generate an action selection output.

[0009] In some implementations, selecting an action to be performed by the agent in response to a current observation using the current combined embedding includes: providing the current combined embedding as input to an action selection neural network to obtain a current action selection output; and using the current action selection output to select an action to be performed by the agent in response to a current observation.

[0010] In some implementations, the current action selection output defines a probability distribution over possible actions the agent should perform. Selecting an action for the agent to perform includes sampling actions from the probability distribution or selecting the action with the highest probability based on the probability distribution.

[0011] In some implementations, for each of a plurality of possible actions the agent is to perform, the current action selection output includes a corresponding Q-value, which is an estimate of the reward the agent would receive for performing the possible action in response to a current observation. Selecting the action the agent is to perform includes selecting the action with the highest Q-value.

[0012] In some implementations, the current action selection output identifies the best possible action to be performed by the agent in response to the current observation, and selecting the action to be performed by the agent includes selecting the best possible action.

[0013] In some implementations, the language encoder model is a recurrent neural network.

[0014] In some implementations, the language encoder model is a bag-of-words encoder.

[0015] In some implementations, the current text string is a natural language instruction for the agent to perform the current task.

[0016] In some implementations, the current observation embedding is the feature matrix of the current observation, and the current text embedding is the feature vector of the current text string.

[0017] In some implementations, combining the current observation embedding and the current text embedding includes: flattening the feature matrix of the current observation; and concatenating the flattened feature matrix with the feature vector of the current text string.

[0018] In some implementations, the current text string is the same for each observation received during the execution of the current task.

[0019] In some implementations, the current text string is different from the previous text string received during the execution of the current task.

[0020] According to a second aspect, a method is provided for training a language encoder model, an observation encoder neural network, and an action selection neural network. The method includes: receiving a current reward as a result of an agent performing a current action in response to a current observation; determining a task-specific update of the current values ​​of the parameters of the action selection neural network by iteratively performing machine learning training techniques to optimize a task-specific objective based on the current reward; and determining the task-specific update of the current values ​​of the parameters of the language encoder model and the observation encoder neural network by backpropagating the gradient of the task-specific objective through the action selection neural network and into the language encoder model and the observation encoder neural network.

[0021] In some implementations, the task-specific objective is to maximize the time-discounted expected reward received in response to actions performed by the agent during the execution of the current task. Iterations of the machine learning training techniques include iterating through reinforcement learning techniques to optimize the task-specific objective using the current reward and the current action selection output.

[0022] In some implementations, the current action selection output also includes a predicted expected reward output, which is an estimate of the time-discounted reward resulting from the current state of the environment. Iterations of reinforcement learning techniques include iterations of performer-evaluator techniques.

[0023] In some implementations, the method further includes determining one or more auxiliary outputs using the current observation embedding, the current text embedding, or both. Auxiliary updates of the current values ​​of the parameters of the language encoder model, the observation encoder neural network, and the action selection neural network are determined by iteratively performing machine learning training techniques to optimize the auxiliary objective based on one or more auxiliary outputs.

[0024] In some implementations, determining one or more auxiliary outputs includes: using a time autoencoder neural network to predict the next observation to be received after the agent performs a selected action, the time autoencoder neural network receiving the embedding of the selected action and the current observation embedding as input.

[0025] In some implementations, determining the auxiliary update includes: obtaining the actual next observation. The update of the current values ​​of the parameters of the observation-encoding neural network is determined by performing gradient descent iterations to minimize the loss between the actual next observation and the predicted next observation.

[0026] In some implementations, the time autoencoder neural network shares one or more parameters with the action selection neural network, and determining the auxiliary update also includes determining the update of the current value of the shared parameters.

[0027] In some implementations, determining one or more auxiliary outputs includes: obtaining data identifying meaningful words, wherein the meaningful words are words in the current text string that influence the agent's performance of the current task. A language prediction neural network, sharing at least some parameters with the observation encoding neural network, processes the current observation to generate a probability distribution over the word vocabulary, where the probability of each word represents the likelihood that the word should be a meaningful word.

[0028] In some implementations, determining the auxiliary update includes: determining an update of the current values ​​of the parameters of the observation-encoding neural network by performing gradient descent iterations to minimize the negative log-likelihood of the probability of the determined meaningful word.

[0029] In some implementations, determining one or more auxiliary outputs further includes: using a reward prediction neural network to process the intermediate output of an action selection neural network, the reward prediction neural network being configured to generate a predicted reward that is an estimate of the reward an agent will receive as a result of performing a current action in response to a current observation.

[0030] In some implementations, determining the auxiliary update includes: determining updates to the current values ​​of the parameters of the language encoder model, the observation encoder neural network, and the action selection neural network by performing gradient descent iterations to minimize the loss between the predicted reward and the current reward.

[0031] In some implementations, the current action selection output also includes a predicted expected reward output, which is an estimate of the time-discounted reward resulting from the current state of the environment. Determining one or more auxiliary outputs also includes: resampling the most recent observations and most recent text embeddings from the replay memory, and processing the most recent observations and most recent text embeddings to generate a predicted expected reward output for the most recent observations and most recent text embeddings.

[0032] In some implementations, determining the auxiliary update includes: determining the update of the current values ​​of the parameters of the language encoder model, the observation encoder neural network, and the action selection neural network by performing an iteration of value function regression using the expected return of the predictions of the most recent observations and the most recent text embeddings.

[0033] According to a third aspect, a system is provided that includes one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the aforementioned method.

[0034] According to the fourth aspect, one or more computer storage media are provided that store instructions, when executed by one or more computers, cause the one or more computers to perform the operations of the aforementioned method.

[0035] According to a fifth aspect, a system is provided that includes one or more computers and one or more storage devices, the storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform corresponding operations of the aforementioned subsystem.

[0036] According to the sixth aspect, one or more computer storage media are provided, which store instructions that, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the aforementioned subsystem.

[0037] According to the seventh aspect, one or more computer storage media are provided that store the corresponding instructions of the aforementioned system.

[0038] According to the eighth aspect, a computational system is provided for learning the relationship between input data of two different modes relating to a public environment. The system includes a first environment neural network module that receives environment input data representing the environment and outputs environment feature data representing features of the environment. The system includes a second task neural network module that receives task input data relating to one or more tasks to be performed with respect to the environment and outputs an embedded representation of the task input data. The system includes a policy definition neural network module that defines a policy for the computational system. The policy definition neural network module is coupled to receive data from the environment neural network module and the task neural network module and outputs action data according to the policy. The action data represents an action to be performed in the environment. The policy defines the action to be performed in the environment in response to the environment input data and the task input data. An action is one of a sequence of actions controlled by a system designed to complete a task. The policy definition neural network module includes a recurrent neural network that defines states in a sequence of states of the computational system.

[0039] The system includes a reinforcement learning training module coupled to an environment neural network module, a task neural network module, and a policy definition neural network module. The reinforcement learning training module trains the environment neural network module, the task neural network module, and the policy definition neural network module in response to reward data representing successful execution of one or more tasks. The system includes at least one auxiliary neural network module, wherein the auxiliary neural network module includes a first set and a second set of weights shared with one or more of the environment neural network module, the task neural network module, and the policy definition neural network module for training the environment neural network module. The auxiliary neural network module is configured to perform unsupervised training using one or both of the environment input data and the task input data.

[0040] In some implementations, the computing system includes a combination module between first and second neural network modules and a policy definition neural network module. The combination module has inputs coupled to the first and second neural network modules to combine embedded representations of environmental feature data and task input data, and is configured to output combined representation data for the policy definition neural network module.

[0041] In some implementations, at least one auxiliary neural network module includes a predictive auxiliary neural network module. The predictive auxiliary neural network module is configured to perform unsupervised training based on a comparison of the current state of the environmental input data with a predicted future state of the environmental input data.

[0042] In some implementations, the prediction-aided neural network module is configured to predict the future state of the environmental input data from the current state of the environmental input data and the action data. The first set of weights includes weights shared with the environmental neural network module and weights shared with the policy definition neural network.

[0043] In some implementations, the prediction auxiliary neural network module includes a time-autoregressive neural network.

[0044] In some implementations, at least one auxiliary neural network module includes a modeling auxiliary neural network module. The modeling auxiliary neural network module is configured for unsupervised training based on the connections between environmental input data and task input data.

[0045] In some implementations, the modeling-aided neural network module is configured to receive environmental input data and output one or more data elements representing task input data. A first set of weights includes weights shared with both the environmental neural network module and the task neural network module. The modeling-aided neural network module is configured for unsupervised training based on comparisons between the task input data and one or more data elements representing the task input data.

[0046] In some implementations, the task neural network module includes a recurrent neural network, and the task input data includes a sequence of data items related to one or more tasks to be performed.

[0047] In some implementations, environmental input data includes image data.

[0048] In some implementations, the task input data includes language data.

[0049] According to the ninth aspect, a non-transitory data carrier is provided, which carries processor control code to implement the aforementioned computing system.

[0050] According to a tenth aspect, a method is provided for training a neural network-based computing system to learn a relationship between input data of two different modes relating to a public environment. The computing system includes: first and second neural network modules that process environment-related first and second mode input data; and a policy neural network module coupled to the first and second neural network modules to output action data depending on the first and second mode input data according to a state-action policy. The method includes training the computing system through reinforcement learning based on a reward defined by a combination of the first and second mode input data. In conjunction with the training through reinforcement learning, the method includes training one or more auxiliary neural network modules through unsupervised learning based on the first and second mode input data. The one or more auxiliary neural network modules share weights with one or more of the first and second neural network modules and the policy neural network module.

[0051] In some implementations, the method includes using one or more auxiliary neural network modules to predict the impact of an action defined by action data on the environment, and training one or more auxiliary neural network modules based on the observed impact of the action.

[0052] In some implementations, the method includes using one of one or more auxiliary neural network modules to model a combination of first and second mode input data.

[0053] In some implementations, the first mode input data includes image data, and the second mode input data includes language data.

[0054] According to the eleventh aspect, a non-transitory data carrier is provided, which carries processor control code to implement the aforementioned method.

[0055] According to a twelfth aspect, an electronic controller trained by reinforcement learning is provided to control a system having multiple states, and for each state, a set of control actions to move from one state to the next. The electronic controller includes a first input to receive state data from the system. The electronic controller includes a second input to receive instruction data defining instructions to be executed by the system. The electronic controller includes a computing system including first and second neural network modules to process data from the first and second mode inputs. The computing system includes a policy neural network module coupled to the first and second neural network modules to output action data based on the state data and instruction data according to a state-action policy. The electronic controller includes at least one auxiliary neural network module, wherein the auxiliary neural network module includes weights shared with one or more of the first, second, and policy neural network modules to train an environment neural network module, and a second set of weights.

[0056] In some implementations, one or both of the strategy neural network module and the first neural network module include a recurrent neural network.

[0057] Specific implementations of the subject matter described in this specification can be carried out to achieve one or more of the following advantages. Systems as described in this specification can be trained to select actions that cause an agent interacting with the environment to execute written instructions specified by a text string. Thus, the system can learn to understand the meaning of natural language by associating natural language text strings with the environment. More specifically, the system can learn to associate components of natural language text strings (e.g., words and phrases) with observations of the environment and sequences of associated actions. Creating systems that understand the meaning of natural language (such as those described in this specification) is a long-standing challenge in the development of artificial intelligence (AI), which is crucial for facilitating human-AI interaction.

[0058] The system described in this specification can be trained based on text strings and observations of the environment (e.g., visual observations) without requiring any prior knowledge of language or the connection between the text strings and observations. Therefore, unlike some conventional systems for computational language learning that rely on rule-based methods and prior knowledge, the system described in this specification acquires semantic knowledge that can be generalized relative to new environments and new text strings. For example, the system described in this specification can use words it already knows to interpret unfamiliar phrases in a text string and successfully execute instructions specified by the text string in an unfamiliar environment. Moreover, as the system described in this specification guides its existing knowledge to enable the acquisition of new knowledge, the rate at which the system learns new words can accelerate over time.

[0059] By eliminating the need for rule-based methods and prior knowledge required by some conventional systems, the system described in this specification can reduce the use of computational resources (e.g., memory) compared to some conventional systems. For example, the system described in this specification acquires knowledge that can be generalized to new environments and new text strings, while some conventional systems would need (e.g., in memory) to store new rules and prior knowledge in order to process data from new environments and new text strings.

[0060] The system described in this specification can be trained using a combination of task-specific parameter updates and auxiliary parameter updates. The task-specific parameter updates are based on the rewards received by the system for progressing through the task, while the auxiliary parameter updates utilize the potentially useful perceptual feedback streams from observations available at each time step. Using auxiliary parameter updates to train the system reduces the amount of computational resources (e.g., memory and computing power) required to train the system, especially when the rewards received by the system are sparse (i.e., few, merely non-zero). For example, if the system receives a non-zero reward upon completion of the task but receives very few non-zero rewards upon completion, the system's rewards may be sparse. Compared to some conventional systems that do not benefit from auxiliary parameter updates, the system described in this specification can be trained to achieve acceptable levels of task performance in fewer training iterations.

[0061] Intelligent agents can be robots, or semi-autonomous or autonomous vehicles. For such devices, the environment can be the real world, in which case observations can be obtained from cameras, laser sensors, or other such devices, and the actions can be real-world tasks, such as navigating through the environment.

[0062] More specifically, a method executed by one or more computers is provided for selecting an action to be performed by an agent interacting with an environment, the method comprising: at each of a plurality of time steps: receiving a current text string in natural language, the current text string expressing information about a current task being performed by the agent; receiving a current observation representing the current state of the environment; processing an input including the current text string and the current observation using a policy neural network to generate an action selection output; and selecting an action to be performed by the agent at the time step based on the action selection output; wherein the policy neural network has been trained end-to-end using reinforcement learning.

[0063] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0064] Figure 1A This is a block diagram of an example action selection system.

[0065] Figure 1B This is a block diagram of an example architecture for an action selection system.

[0066] Figure 2 This is a flowchart of an example process for selecting the action to be performed by the agent.

[0067] Figure 3 This is a flowchart of an example process for training an action selection system using task-specific parameter updates.

[0068] Figure 4 This is a flowchart of an example process for training an action selection system using auxiliary parameter updates.

[0069] Similar reference numerals and labels in the various figures indicate similar elements. Detailed Implementation

[0070] Figure 1A An example action selection system 100 is shown. Action selection system 100 is an example of a system that implements a computer program on one or more computers in one or more locations, wherein the systems, components and techniques described below are implemented.

[0071] The action selection system 100 selects an action 102 to be performed by an agent 104 interacting with the environment 106 at each of multiple time steps. To enable the agent 104 to interact with the environment 106, at each time step, the system 100 receives input and selects an action 102 to be performed by the agent 104 in response to the received input. Typically, the input includes data characterizing the current state of the environment 106 (e.g., an image of the environment 106) and data of another modality (e.g., a text string 108) expressing some information about the task currently being performed by the agent 104 (or the task to be performed by the agent 104). In this specification, the data characterizing the state of the environment 106 will be referred to as observation 110.

[0072] The system 100 described herein is broadly applicable and is not limited to a particular implementation. However, for illustrative purposes, a few example implementations are described below.

[0073] In some implementations, environment 106 is a real-world environment, and agent 104 is a mechanical agent that interacts with the real-world environment. For example, agent 104 may be a robot that interacts with environment 106 to perform a specific task. As another example, agent 104 may be an autonomous or semi-autonomous vehicle that navigates through environment 106. In these implementations, action 102 may be a control input that controls the robot or autonomous vehicle. In some of these implementations, observation 110 may be generated by or derived from sensors of agent 104. For example, observation 110 may be captured by a camera of agent 104. As another example, observation 110 may be derived from data captured by a laser sensor of agent 104. As another example, observation 110 may be a hyperspectral image captured by a hyperspectral sensor of agent 104.

[0074] In some implementations, environment 106 may be a simulated environment, and agent 104 may be implemented as one or more computer programs that interact with the simulated environment. For example, simulated environment 106 may be a video game, and agent 104 may be a simulated user playing the video game. As another example, simulated environment 106 may be a motion simulation environment, such as a driving simulator or flight simulator, and agent 104 may be a simulated vehicle navigating through the motion simulation environment. In these implementations, action 102 may be a control input that controls the simulated user or the simulated vehicle.

[0075] As previously described, the input received by system 100 includes data in another modality (e.g., a modality different from that of observation 110) that expresses some information about the task currently being performed by agent 104 (or the task to be performed by agent 104). For example, the other modal data could be instructions in natural language (e.g., English) for agent 104 to perform the task. In a particular example, the task could be for agent 104 to navigate to a specific location in environment 106 (e.g., in this case, the instruction might be: "Navigate to school"), or to locate a specific object in environment 106 (e.g., in this case, the instruction might be: "Find the red object"). The other modal data can be submitted by a user of system 100, for example, via a keyboard or microphone. The user of system 100 could be, for example, a driver of autonomous vehicle agent 104 or an operator of robotic agent 104. While system 100 is described herein as receiving other modal data in the form of a text string 108, more generally, the other modal data can have any suitable format. For example, the other modal data could be audio data expressing the speech of the user of system 100.

[0076] In some implementations, the text string 108 received by system 100 is the same at every time step during the execution of the current task. For example, at each time step during the execution of the current task, the text string 108 may be a natural language instruction for agent 104 to perform the current task. Typically, when the text string 108 received by system 100 is the same at every time step, system 100 can maintain the text string 108 (e.g., in memory) and does not require the user of system 100 to repeatedly provide the same text string 108 across multiple time steps. In some implementations, the text string 108 received by system 100 may differ between time steps during the execution of the current task. For example, at a later time step during the execution of the current task, the text string 108 may be an instruction to perform the current task, wherein the instruction specifies more details about how the task should be performed than at earlier time steps during the execution of the current task.

[0077] System 100 processes the current observation 110 using observation encoder neural network 112 according to the current values ​​of the observation encoder neural network parameters to generate observation embedding 114. In some embodiments, the current observation 110 is an image, the observation encoder neural network 112 is a convolutional neural network, and the current observation embedding 114 is a feature matrix describing the current observation 110. Typically, an embedding refers to a numerical representation of the input, such as a vector, matrix, or other ordered set of values.

[0078] System 100 uses a language encoder model 116 (also referred to as a "language encoder module") to process the representation of the current text string 108 to generate a current text embedding 118. For example, the representation of the current text string 108 can be a sequence of numerical representations, where each numerical representation represents a corresponding character (or word) from a dictionary of possible characters (or words), and the current text embedding 118 can be a feature vector representing the current text string 108. In some implementations, the language encoder model 116 is a recurrent neural network (e.g., a Long Short-Term Memory (LSTM) network) configured to process the current text string 108 over a series of internal time steps to generate the current text embedding 118. In some implementations, particularly when the order of words in the current text string 108 is not important, the language encoder model 116 is a bag-of-words language encoder. A bag-of-words language encoder can generate the current text embedding 118 by summing the numerical representations of each character (or word) in the current text string 108.

[0079] At each time step, system 100 combines the current observation embedding 114 and the current text embedding 118 to determine a current combined embedding 120, and uses the current combined embedding 120 to select an action 102 for agent 104 to perform in response to the current observation 110. For example, system 100 may process the current combined embedding 120 using action selection neural network 122 according to the current values ​​of the action selection neural network parameters to generate an action selection output 124. In this example, system 100 may select the action 102 to be performed at that time step based on the action selection output 124. The action selection neural network 122 may be implemented as, for example, a recurrent neural network (e.g., LSTM) or a feedforward neural network (e.g., a multilayer perceptron).

[0080] In some implementations, the action selection output 124 is defined as a probability distribution over actions in a set of possible actions that can be performed by agent 104. For example, the action selection output 124 may include the corresponding numerical likelihood value for each action in the set of possible actions. As another example, the action selection output 124 may include parameter values ​​(e.g., mean and standard deviation values) of a parametric probability distribution (e.g., a normal distribution) defined over the set of possible actions. In some of these implementations, system 100 selects action 102 to be performed by agent 104 by sampling actions from the set of possible actions based on the probability distribution. In some of these implementations, the system selects the action with the highest probability from the set of possible actions as action 102 to be performed by agent 104. In other implementations of these implementations, the system selects the action to be performed by the agent by sampling from the probability distribution.

[0081] In some implementations, the action selection output 124 identifies the best possible action that agent 104 should perform in response to the current observation 110, and system 100 selects action 102 as the best possible action. For example, if agent 104 is a robotic agent, the action selection output 124 may identify the best possible torque to be applied to the joints of agent 104.

[0082] In some implementations, the action selection output 124 includes the corresponding Q-value for each action in the set of possible actions. The Q-value of an action is an estimate of the return that the system 100 will receive if agent 104 performs the action in response to the current observation 110. A return refers to a cumulative measure of the reward received by the system 100 as agent 104 interacts with the environment 106 over multiple time steps. For example, a return could refer to the long-term time-discounted sum of rewards received by the system 100. In some of these implementations, the system selects the action 102 that agent 104 is to perform at that time step as the action with the highest Q-value.

[0083] At each time step, system 100 can receive a reward 126 based on the current state of environment 106 and the current action 102 of agent 104. Typically, reward 126 is represented as one or more numerical values. Reward 126 can be based on any event or aspect of environment 106. For example, a reward could indicate whether agent 104 has successfully performed the task specified by text string 108.

[0084] System 100 includes a training engine 128 configured to train system 100 over multiple training iterations. Training engine 128 trains system 100 by iteratively (i.e., in each training iteration) adjusting the current parameter values ​​of one or more of the action selection neural network 122, the observation encoder neural network 112, and the language encoder model 116. Training engine 128 enables system 100 to select actions 102 that allow agent 104 to more effectively perform the task specified by text string 108. Thus, training engine 128 enables system 100 to learn to understand the meaning of natural language by relating natural language text string 108 to environment 106. More specifically, training engine 128 enables system 100 to learn to correlate components of natural language text string 108 (e.g., words and phrases) with observations 110 and associated action sequences 102 of environment 106.

[0085] In each of the multiple training iterations, the training engine 128 determines task-specific parameter updates 130 and applies them to adjust the current parameter values ​​of one or more of the action selection neural network 122, the observation encoder neural network 112, and the language encoder model 116. (See reference...) Figure 3 Further described, the training engine 128 can determine task-specific parameter updates 130 by performing iterative machine learning training techniques to optimize a task-specific objective based on the system-received reward 126. For example, the task-specific objective could be maximizing the expected reward received in response to the action 102 performed by agent 104 during the execution of the current task, with a time discount. The machine learning training technique could be a reinforcement learning technique (e.g., actor-critic reinforcement learning) to optimize the task-specific objective using the current reward 126 and the current action selection output 124. The machine learning training technique could include backpropagating the gradient of the task-specific objective through the action selection neural network 122 and into the language encoder model 116 and the observation encoder neural network 112.

[0086] In addition to training system 100 using task-specific parameter updates based on reward 126 (which may be sparse), training engine 128 can also utilize potentially useful perceptual feedback streams from observations 110 available at each time step as agent 104 explores environment 106. At each of multiple training iterations, training engine 128 can determine auxiliary parameter updates 132 and apply them to adjust the current parameter values ​​of one or more of the action selection neural network 122, observation encoder neural network 112, and language encoder model 116. (See reference...) Figure 4 Further described, the training engine can determine auxiliary parameter updates 132 by performing iterative machine learning training techniques to optimize the auxiliary objective based on one or more auxiliary outputs 134. System 100 can use one or more auxiliary neural networks 136 (and in some cases, an action selection neural network 122) to generate the auxiliary outputs 134. The machine learning training techniques may include backpropagating the gradients of the auxiliary objective to the action selection neural network 122, the language encoder model 116, and the observation encoder neural network 112.

[0087] The auxiliary neural network 136 can process inputs including the current observation embedding 114, the current text embedding 118, or both, to generate an auxiliary output 134 involving prediction or modeling of aspects of the environment 106. For example, the auxiliary neural network 136 may include a temporal autoencoder neural network configured to generate an auxiliary output including a prediction of the next observation 110 to be received by agent 104 after performing the current action 102 at the current time step. As another example, the auxiliary neural network 136 may include a language prediction neural network configured to generate an auxiliary output 134 defining predictions of meaningful words included in the current text string 108. As yet another example, the auxiliary neural network 136 may include a reward prediction neural network configured to generate an auxiliary output 134 including a predicted reward, which is an estimate of the reward that agent 104 will receive due to performing the current action 102 at the current time step. As another example, the action selection output 124 of the action selection neural network 122 may include an auxiliary output that defines a predicted expected reward output, which is an estimate of the time-discounted reward due to the current state of the environment.

[0088] Figure 1B It is an action selection system (e.g., Figure 1A A block diagram of an example architecture 138 for an action selection system 100.

[0089] In example architecture 138, observation 110 is image 140, observation encoder neural network 112 is convolutional neural network, and language encoder model 116 is LSTM recurrent neural network. Action selection neural network 122 is also LSTM recurrent neural network, which is configured at each time step to generate an output that defines the action 102 to be performed by agent 104 and an estimate of the value 142 of the current state, i.e., the time-discounted reward generated due to the environment 106 being in the current state.

[0090] The temporal autoencoder neural network 144 is a deconvolutional neural network configured to process inputs including the embedding of the current action 102 and the current observation embedding 114 to generate an auxiliary output including the next observation prediction 146. The next observation prediction 146 is a prediction of the image 140 to be received by the agent 104 after performing the current action 102 at the current time step.

[0091] The language prediction neural network 148 is configured to process inputs including the current image 140 and the intermediate output of the language encoder model 116 to generate an auxiliary output that defines the prediction of meaningful words 150 contained in the text string 108. Meaningful words are those that influence the agent 104's performance on the current task. The language prediction neural network 148 shares some parameters with the observation encoder neural network 112.

[0092] The reward prediction neural network 152 is configured to process inputs including the intermediate output of the action selection neural network 122 to generate an auxiliary output including a predicted reward 154. The predicted reward 154 is an estimate of the reward that agent 104 will receive as a result of performing the current action 102 at the current time step.

[0093] System 100 generates an auxiliary value 156, which is output by resampling the most recently observed embedding and the most recently written embedding from the replay memory and processing the most recently observed embedding and the most recently written embedding using an action selection neural network 122 to generate the auxiliary value 156. The auxiliary value 156 is an estimate of the value of the state represented by the most recently observed embedding and the most recently written embedding.

[0094] Training engine 128, based on the rewards received by system 100, uses executor-evaluator reinforcement learning techniques to determine task-specific parameter updates for system 100, such as referencing... Figure 3 Further described. The training engine 128 determines the auxiliary parameter updates of the system 100 based on the next observation prediction 146, the predicted meaningful word 150, the prediction reward 154, and the auxiliary value 156, as will be referred to Figure 4 Further description.

[0095] Figure 2 This is a flowchart of an example process for selecting an action (i.e., at a given time step) to be performed by an agent. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to this specification, such as... Figure 1A The action selection system 100 can perform processing 200.

[0096] The system receives a current text string (202) that expresses some information about the task the agent is currently performing (or the task the agent is about to perform). For example, the current text string could be instructions in natural language (e.g., English) for the agent to perform the task.

[0097] The system obtains the current text embedding (204) of the current text string. More specifically, the system processes the representation of the current text string using a language encoder model to generate the current text embedding. For example, the representation of the current text string can be a sequence of numerical representations, where each numerical representation represents a corresponding character (or word) from a dictionary of possible characters (or words), and the current text embedding can be a feature vector representing the current text string. In some implementations, the language encoder model is a recurrent neural network (e.g., a Long Short-Term Memory network (LSTM)) configured to process the current text string over a series of internal time steps to generate the current text embedding. In some implementations, particularly when the order of words in the current text string is not important, the language encoder model is a bag-of-words language encoder. A bag-of-words language encoder can generate the current text embedding by summing the numerical representations of each character (or word) in the current text string.

[0098] The system receives a current observation (206) characterizing the current state of the environment. This observation can be generated by or derived from the agent's sensors. For example, the observation can be captured by the agent's camera. As another example, the observation can be derived from data captured by the agent's laser sensor. As yet another example, the observation can be a hyperspectral image captured by the agent's hyperspectral sensor.

[0099] The system obtains the current observation embedding (208). More specifically, the system processes the current observation using the observation encoder neural network according to the current values ​​of the observation encoder neural network parameters to generate the current observation embedding. In some implementations, the current observation is an image, the observation encoder neural network is a convolutional neural network, and the current observation embedding is a feature matrix describing the current observation.

[0100] The system combines the current observation embedding and the current text embedding to obtain the current combined embedding (210). For example, the system can flatten the feature matrix of the current observation (as determined in step 208) and concatenate the flattened feature matrix with the feature vector of the current text string (as determined in step 204). Flattening the matrix means transforming the matrix into a vector by concatenating the components of the matrix into vectors in a predetermined order.

[0101] The system uses the current combined embedding to select the action to be performed by the agent (212). For example, the system can use the action selection neural network to process the current combined embedding according to the current values ​​of the action selection neural network parameters to generate an action selection output. In this example, the system can select the action to be performed at the time step based on the action selection output.

[0102] In some implementations, the action selection output defines a probability distribution over actions in a set of possible actions that can be performed by the agent. For example, the action selection output may include the corresponding numerical likelihood value for each action in the set of possible actions. As another example, the action selection output may include parameter values ​​(e.g., mean and standard deviation values) of a parametric probability distribution (e.g., a normal distribution) defined over the set of possible actions. In some of these implementations, the system selects the action to be performed by the agent by sampling actions from the set of possible actions based on the probability distribution. In some of these implementations, the system selects the action with the highest probability from the set of possible actions as the action to be performed by the agent, based on the probability distribution.

[0103] In some implementations, the action selection output identifies the best possible action the agent should perform in response to the current observation, and the system selects the action as the best possible action. For example, if the agent is a robotic agent, the action selection output may identify the best possible torque to be applied to the agent's joints.

[0104] In some implementations, the action selection output includes the corresponding Q-value for each action in the set of possible actions. The Q-value of an action is an estimate of the reward the system would receive if the agent performed the action in response to the current observation. The reward refers to a cumulative measure of the rewards received by the system as the agent interacts with the environment over multiple time steps. For example, the reward could refer to a long-term time-discounted reward received by the system. In some of these implementations, the system selects the action to be performed by the agent at a given time step as the action with the highest Q-value.

[0105] In some implementations, the system selects an action to be performed by the agent according to an exploration strategy. Typically, selecting an action according to an exploration strategy means selecting an action that differs from the action to be selected based on (e.g., as defined by the current parameter values ​​of the action selection neural network) the current action selection strategy. For example, the system may select an action to be performed by the agent according to a greedy exploration strategy of probability ϵ. In this example, the system may select an action to be performed by the agent based on an action selection output with probability 1-ϵ, and may also randomly select an action to be performed by the agent with probability ϵ (where ϵ is a number between 0 and 1).

[0106] Figure 3 This is a flowchart of an example process for training an action selection system using task-specific parameter updates (i.e., with a given training iteration). For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to this specification, such as... Figure 1AThe action selection system 100 can perform processing 300.

[0107] The system receives a reward (302). Typically, the reward is represented as one or more numerical values. The reward can be based on any event or aspect of the environment. For example, a reward could indicate whether the agent has successfully performed a task specified by a text string. In some implementations, the system receives the current reward as a result of the agent performing a current action in response to a current observation, as referenced. Figure 2 As described. In some implementations, the system obtains data from a replay memory, which includes previous rewards for previous time steps, previous actions performed by the agent, and observations characterizing the previous state of the environment. A replay memory refers to an external memory (e.g., a logical data storage area or a physical data storage device) in which the system can store data (e.g., observations, selected actions, rewards, etc.) at each time step.

[0108] The system determines task-specific parameter updates by iteratively performing machine learning training techniques to optimize a task-specific objective based on the rewards received by the system (304). For example, the task-specific objective could be to maximize the time-discounted expected reward received in response to actions performed by the agent during the execution of the current task. In this example, the time-discounted expected reward can be given as follows:

[0109]

[0110] in It is the expectation operator relative to the system's action selection policy π, λ is the discount factor (usually a number between 0 and 1), t is the time step index, and This is the reward received by the system at time step t. Machine learning training techniques can be reinforcement learning techniques used to optimize task-specific objectives. For example, reinforcement learning techniques can be policy gradient techniques, one-step or n-step Q-learning techniques, or actor-evaluator techniques. When the reinforcement learning technique is an actor-evaluator technique, the action selection output of the action selection neural network can include an estimate of the current state value, i.e., the time-discounted reward resulting from the environment being in the current state.

[0111] The system can determine task-specific parameter updates by determining the gradient of a task-specific objective (e.g., as defined by equation (1)) with respect to the parameters of the action selection neural network, the language encoder model, and the observation encoder neural network. The system can use, for example, backpropagation techniques to determine the gradient of the task-specific objective. The task-specific parameter updates can be represented as vectors, matrices, or numerical values, where each numerical value corresponds to a different parameter of the action selection neural network, the language encoder model, or the observation encoder neural network.

[0112] The system uses task-specific parameter updates to update the current values ​​(306) of the parameters of the action selection neural network, the language encoder model, and the observation encoder neural network. For a given set of parameters... The system can update the current value of the parameter in the following ways:

[0113]

[0114] in It is a positive learning rate hyperparameter (i.e., a system hyperparameter that controls how quickly its value can change during training), and It is a task-specific objective relative to parameters (e.g., as determined in step 304). The gradient.

[0115] Figure 4 This is a flowchart of an example process for updating (i.e., at a given training iteration) the training action selection system using auxiliary parameters. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to this specification, such as... Figure 1A The action selection system 100 can perform processing 400.

[0116] The system determines one or more auxiliary outputs (402). The system can use one or more auxiliary neural networks, and in some cases, an action selection neural network, to generate the auxiliary outputs.

[0117] In some implementations, the auxiliary neural network includes a temporal autoencoder neural network. The temporal autoencoder neural network is configured to process inputs including an embedding of the current action and an embedding of the current observation to generate an output including a prediction of the next observation to be received by the agent after performing the current action at the current time step. For example, if the observation is an image of the environment (e.g., an RGB image), the temporal autoencoder neural network generates a predicted image of the environment after the agent performs the current action at the current time step.

[0118] In some implementations, the temporal autoencoder neural network and the action selection neural network share one or more parameters. Two neural networks are said to share a parameter if the parameter values ​​are the same for both networks, and changing the parameter value in one neural network (e.g., during training) will also cause the parameter value in the other neural network to change. In a particular example, the temporal autoencoder neural network can represent intermediate representations. Determined as:

[0119]

[0120] in , and It is a parameter matrix, ⊙ indicates element-wise multiplication, y is the current observation embedding, and It is the embedding of the current action, where the parameter matrix It shares its parameters with the action selection neural network. The time autoencoder neural network can use a deconvolutional neural network to process intermediate representations. This is used to generate a prediction for the next observation.

[0121] In some implementations, the auxiliary neural network includes a language prediction neural network. The language prediction neural network may share some parameters with the observation encoding neural network and is configured to process inputs including the intermediate outputs of the current observation and the language encoder model to generate an auxiliary output that defines predictions of meaningful words contained in the current text string. Meaningful words are words that influence the agent's performance of the current task. For example, if the current text string specifies the current task as "navigate to the pharmacy," then the word "pharmacy" could be a meaningful word. The auxiliary output generated by the language prediction neural network can be a probability distribution over the word vocabulary, where the probability of each word represents the likelihood that the word should be a meaningful word. In some cases, the auxiliary output of the language prediction neural network can be interpreted as a prediction defining the word that best describes the current observation.

[0122] In some implementations, the auxiliary neural network includes a reward prediction neural network. The reward prediction neural network is configured to process inputs including the intermediate output of an action selection neural network to generate an auxiliary output including a predicted reward, which is an estimate of the reward the agent will receive as a result of performing the current action at the current time step.

[0123] In some implementations, the action selection output of the action selection neural network includes a value (i.e., the predicted expected reward) output, which is an estimate of the time-discounted reward resulting from the environment being in the current state. In some of these implementations, the system generates an auxiliary value (i.e., the predicted expected reward) output by resampling the most recently observed embeddings and most recently written embeddings from the replay memory and processing these embeddings using the action selection neural network. The auxiliary value output is an estimate of the value of the state represented by the most recently observed embeddings and most recently written embeddings. The most recently observed embeddings and most recently written embeddings refer to the observation embeddings and text embeddings generated at previous time steps. The system can resample the most recently observed embeddings and most recently written embeddings from the replay memory by obtaining the observation embeddings and text embeddings corresponding to randomly sampled previous time steps.

[0124] The system determines auxiliary parameter updates by performing iterative machine learning training techniques to optimize auxiliary targets based on one or more auxiliary outputs (404).

[0125] In some implementations, the auxiliary output includes the output of the temporal autoencoder neural network, including a prediction of the next observation to be received by the agent after performing the current action at the current time step (as described in step 402). In some of these implementations, the system obtains the actual next observation and determines auxiliary parameter updates for the current values ​​of the parameters of the observation encoding neural network by performing iterative gradient descent to minimize the loss between the actual next observation and the predicted next observation. For example, the loss could be a mean squared error (MSE) loss. In some of these implementations, the temporal autoencoder neural network shares one or more parameters with the action selection neural network, and the system determines auxiliary parameter updates relative to the current values ​​of the shared parameters.

[0126] In some implementations, the auxiliary output includes the output of the language prediction neural network, comprising a probability distribution over word lexicons, where the probability of each word represents the likelihood that the word is a meaningful word (as described in step 402). In some of these implementations, the system obtains data identifying meaningful words and determines auxiliary parameter updates for the current values ​​of the parameters of the observation encoding neural network by performing gradient descent iterations to minimize the negative log-likelihood of the probabilities of the determined meaningful words.

[0127] In some implementations, the auxiliary output includes the output of the reward prediction neural network, comprising a predicted reward that is an estimate of the reward the agent will receive as a result of performing the current action at the current time step (as described in step 402). In some of these implementations, the system determines auxiliary updates to the parameters of the language encoder model, the observation encoder neural network, and the action selection neural network by performing iterations of gradient descent to minimize the loss between the predicted reward and the current reward. For example, the loss could be a squared error loss.

[0128] In some implementations, the auxiliary output includes the expected reward output of the prediction generated by the action selection neural network by processing the most recently observed embedding and the most recently textual embedding. In some of these implementations, the system determines auxiliary updates to the current values ​​of the parameters of the language encoder model, the observation encoder neural network, and the action selection neural network by performing an iteration of value function regression using the expected reward of the prediction of the most recently observed and most recently textual embeddings.

[0129] When a system determines multiple auxiliary parameter updates (e.g., based on auxiliary outputs generated by multiple auxiliary neural networks), the system can combine these multiple auxiliary parameter updates to determine the final auxiliary parameter update. For example, the system can combine multiple auxiliary parameter updates through a weighted linear combination, where the weights of the linear combination are system hyperparameters.

[0130] The system uses auxiliary parameter updates to update the current values ​​of the parameters of the action selection neural network, the language encoder model, and the observation encoder neural network (406). Refer to step 306 for an example process for updating the current values ​​of the system parameters.

[0131] This specification uses the term "configured" in relation to system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, this means that software, firmware, hardware, or a combination thereof have been installed on the system, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a specific operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0132] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more computer program instruction modules encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions may be encoded on artificially generated propagated signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0133] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0134] A computer program can also be referred to or described as a program, software, software application, app, module, software module, script, or code. It can be written in any programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data, for example, as one or more scripts stored in a markup language document, as a single file dedicated to the program in question, or as a collection of files that, for example, store one or more modules, subroutines, or code portions. A computer program can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected via a data communication network.

[0135] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0136] The processing and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processing and logic flows can also be executed by special-purpose logic circuitry (such as an FPGA or ASIC) or a combination of special-purpose logic circuitry and one or more programmable computers.

[0137] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include, or be operatively coupled to, receiving data from or transferring data to one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks. However, such devices are not essential for a computer. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device (e.g., a Universal Serial Bus (USB) flash drive).

[0138] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0139] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to and from the device used by the user, for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user as replies.

[0140] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, namely inference and workloads.

[0141] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0142] Embodiments of the subject matter described in this specification can be implemented in a computing system comprising: back-end components, such as a data server; or middleware components, such as an application server; or front-end components, such as a client computer having a graphical user interface, web browser, or application through which a user can interact with embodiments of the subject matter described in this specification; or any combination of one or more such back-end, middleware, or front-end components. Components of the system can be interconnected via digital data communication through any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0143] A computing system may include clients and servers. Clients and servers are typically geographically separated and interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from the user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0144] While this specification contains numerous specific details of implementation, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features described herein in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0145] Similarly, although operations are depicted in a specific order in the drawings and described in the claims, this should not be construed as requiring these operations to be performed in the specific order or successive order shown, or to perform all the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0146] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method executed by one or more computers for selecting an action to be performed by an intelligent agent interacting with a real-world environment, wherein the intelligent agent is a robot, or a semi-autonomous or autonomous vehicle, the method comprising: At each of the multiple time steps: Receive a current text string in natural language, which expresses information about the current task being performed by the agent; The agent receives current observations from its sensors that characterize the current state of the real-world environment, where the current observations are images of the real-world environment. The policy neural network is used to process the input, which includes the current text string and the current observation, to generate an action selection output, wherein the action is the control input that controls the agent; The process includes: Using a policy neural network, the current text string and the current observation are combined to generate a combined embedding based on the values ​​of the policy neural network parameter set; and An action selection output is generated based on the combined embedding using a policy neural network and the values ​​of the policy neural network parameter set; and Based on the action selection output, select the action that the agent should perform at the time step; The policy neural network has already been trained end-to-end using reinforcement learning. The method further includes: using an auxiliary neural network to generate a predicted image of the real-world environment to be received after the agent performs a selected action as the next current observation, the auxiliary neural network receiving the embedding of the selected action and the current observation embedding as input.

2. The method according to claim 1, further comprising: At each of the plurality of time steps, the current reward is received as a result of the agent performing an action in response to the current observation; as well as Based on the rewards received at the multiple time steps, a policy neural network is trained end-to-end using reinforcement learning.

3. The method of claim 1, wherein generating the combined embedding by combining the current text string and the current observation based on the values ​​of the policy neural network parameter set via a policy neural network comprises: A language encoder model using a policy neural network is used to process the current text string to generate the current text embedding of the current text string; The observation encoder neural network uses a policy neural network to process the current observation to generate the current observation embedding; as well as The current observation embedding and the current text embedding are combined to generate the combined embedding.

4. The method according to claim 3, wherein generating an action selection output based on the combined embedding according to the values ​​of the policy neural network parameter set via the policy neural network comprises: The combined embeddings are processed using a policy neural network and an action selection neural network to generate action selection outputs.

5. The method according to claim 3, wherein, The language encoder model is a recurrent neural network.

6. The method according to claim 3, wherein, The language encoder model is a bag-of-words encoder.

7. The method according to claim 3, wherein, The current observation embedding is the feature matrix of the current observation, and the current text embedding is the feature vector of the current text string.

8. The method of claim 7, wherein combining the current observation embedding and the current text embedding comprises: Flatten the currently observed feature matrix; as well as Concatenate the flattened feature matrix with the feature vector of the current text string.

9. The method of claim 1, wherein in each of the plurality of time steps, the current text string is a natural language instruction for the agent to perform the current task.

10. The method of claim 1, wherein in each of the plurality of time steps: The action selection output defines the probability distribution of possible actions the agent should perform; and The actions to be performed by the agent include: Sampling actions from the probability distribution or selecting the action with the highest probability based on the probability distribution.

11. The method of claim 1, wherein in each of the plurality of time steps: For each of a plurality of possible actions to be performed by the agent, the action selection output includes a corresponding Q-value, which is an estimate of the reward resulting from the agent performing the possible action in response to the current observation; and The actions to be performed by the agent include: Choose the action with the highest Q value.

12. The method of claim 1, wherein in each of the plurality of time steps: Action selection outputs the best possible action that the agent should perform in response to the current observation; and The actions to be performed by the agent include: Choose the best possible action.

13. The method according to claim 1, wherein, For each observation received during the execution of the current task, the current text string is the same.

14. The method according to claim 1, wherein, The current text string is different from the previous text string received during the execution of the current task.

15. A system comprising: One or more computers; as well as One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for selecting actions to be performed by an intelligent agent interacting with a real-world environment, wherein the intelligent agent is a robot, or a semi-autonomous or autonomous vehicle, and the operations include: At each of the multiple time steps: Receive a current text string in natural language, which expresses information about the current task being performed by the agent; The agent receives current observations from its sensors that characterize the current state of the real-world environment, where the current observations are images of the real-world environment. The policy neural network is used to process the input, which includes the current text string and the current observation, to generate an action selection output, wherein the action is the control input that controls the agent; The process includes: Using a policy neural network, the current text string and the current observation are combined to generate a combined embedding based on the values ​​of the policy neural network parameter set; and An action selection output is generated based on the combined embedding using a policy neural network and the values ​​of the policy neural network parameter set; and Based on the action selection output, select the action that the agent should perform at the time step; The policy neural network has already been trained end-to-end using reinforcement learning. The operation further includes: using an auxiliary neural network to generate a predicted image of the real-world environment to be received after the agent performs the selected action as the next current observation, the auxiliary neural network receiving the embedding of the selected action and the current observation embedding as input.

16. A non-transitory computer storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform operations for selecting actions to be performed by an intelligent agent interacting with a real-world environment, wherein the intelligent agent is a robot, or a semi-autonomous or autonomous vehicle, and the operations include: At each of the multiple time steps: Receive a current text string in natural language, which expresses information about the current task being performed by the agent; The agent receives current observations from its sensors that characterize the current state of the real-world environment, where the current observations are images of the real-world environment. The policy neural network is used to process the input, which includes the current text string and the current observation, to generate an action selection output, wherein the action is the control input that controls the agent; The process includes: Using a policy neural network, the current text string and the current observation are combined to generate a combined embedding based on the values ​​of the policy neural network parameter set; and An action selection output is generated based on the combined embedding using a policy neural network and the values ​​of the policy neural network parameter set; and Based on the action selection output, select the action that the agent should perform at the time step; The policy neural network has already been trained end-to-end using reinforcement learning. The operation further includes: using an auxiliary neural network to generate a predicted image of the real-world environment to be received after the agent performs the selected action as the next current observation, the auxiliary neural network receiving the embedding of the selected action and the current observation embedding as input.

17. The non-transitory computer storage medium of claim 16, wherein the operation further comprises: At each of the plurality of time steps, the current reward is received as a result of the agent performing an action in response to the current observation; as well as Based on the rewards received at the multiple time steps, a policy neural network is trained end-to-end using reinforcement learning.

18. The non-transitory computer storage medium of claim 16, wherein combining the current text string and the current observation to generate the combined embedding by means of a policy neural network, based on the values ​​of a set of policy neural network parameters, comprises: A language encoder model using a policy neural network is used to process the current text string to generate the current text embedding of the current text string; The observation encoder neural network uses a policy neural network to process the current observation to generate the current observation embedding; as well as The current observation embedding and the current text embedding are combined to generate the combined embedding.

19. The non-transitory computer storage medium of claim 17, wherein generating an action selection output based on the combined embedding, according to the values ​​of the policy neural network parameter set, via a policy neural network, includes: The combined embeddings are processed using a policy neural network and an action selection neural network to generate action selection outputs.

20. The non-transitory computer storage medium according to claim 18, wherein, The language encoder model is a recurrent neural network.

Images