Deep reinforcement learning with fast-update recurrent neural networks and slow-update recurrent neural networks

Abstract

Methods, systems, and apparatuses for reinforcement learning include computer programs encoded on a computer storage medium. One method involves using both a slow-updating recurrent neural network and a fast-updating recurrent neural network to select an action to be performed by an agent, the fast-updating recurrent neural network receiving fast-updating inputs including the hidden states of the slow-updating recurrent neural network.

Description

Technical Field

[0001] The present disclosure relates to reinforcement learning, and more particularly, to deep reinforcement learning using fast-update recurrent neural networks and slow-update recurrent neural networks. Background Art

[0002] This specification relates to reinforcement learning.

[0003] In a reinforcement learning system, an agent interacts with an environment by performing actions selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.

[0004] Some reinforcement learning systems select actions to be performed by an agent in response to receiving a given observation based on the output of a neural network.

[0005] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks are deep neural networks that include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output based on the received inputs according to the current values ​​of the corresponding set of parameters. Summary of the invention

[0006] This specification generally describes a reinforcement learning system that selects actions to be performed by a reinforcement learning agent that interacts with an environment.

[0007] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0008] Various aspects related to automatically identifying time-extended behavioral patterns of agents interacting with an environment can facilitate evaluation of the performance of an agent by analyzing the behavior of the agent, including a neural network system. More specifically, the described methods and systems are able to identify high-level behavioral patterns that may otherwise be obscured by the complexity of the system. This in turn enables humans to evaluate the agent based on these patterns, such as determining how efficiently the agent is achieving its goals and the techniques it is using. Therefore, these techniques can represent very complex internal states of the agent in an understandable manner, which helps evaluate and thus help implement control of real-world or simulated systems by autonomous or semi-autonomous agents as described above. For example, the described methods and systems can help humans understand how or why an agent behaves when controlling a system, which can help tailor the agent to a specific task.

[0009] Certain described aspects relate to using a temporal hierarchical neural network system to select actions. These aspects allow the action selected at each time step to be consistent with the agent's long-term plan. The agent can then achieve improved performance on tasks that require selecting actions depending on data received in observations at a large number of time steps before the current time step. Because the neural network is jointly trained, performance improvements can be achieved without excessively increasing the amount of computational resources consumed during recurrent neural network training. By using a temporal hierarchical neural network system to generate latent variables and then adjusting action selection on the latent variables, the system can effectively perform tasks even when the received observations do not fully characterize the true state of the environment, that is, when the true state of the environment is unclear even considering all the observations received by the agent so far. For example, many tasks include multiple agents operating independently in the environment, such as multiple autonomous vehicles or multiple robots. In these tasks, the observations received by a single agent cannot fully characterize the true state of the environment because the agent does not know the control policies of other agents or whether these control policies will change over time. Therefore, there is inherent uncertainty because the agent cannot know how other agents will behave or react to changes in the state of the environment. The system can account for this uncertainty by using a temporal hierarchical neural network to sample latent variables from a posterior distribution that depends on both long-term and short-term data received from the environment.

[0010] Certain described aspects allow a system to learn rewards for neural network training. These aspects allow the system to efficiently train a neural network, i.e., train a neural network so that it can be used to cause an agent to have acceptable performance on a reinforcement learning task, even when rewards associated with task performance are very sparse and very delayed. In particular, the system can leverage additional information that can be extracted from the environment to generate an improved reward signal, even when data specifying how the extracted data is related to task performance is not specified before training begins.

[0011] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An example reinforcement learning system is shown.

[0013] Figure 2 The processing performed by the reinforcement learning system at a given time step is shown.

[0014] Figure 3A and Figure 3B An example architecture of a neural network used by a reinforcement learning system is shown.

[0015] Figure 4 is a flow diagram of an example process for selecting an action to be performed by an agent.

[0016] Figure 5 is a flow chart of an example process for updating parameter values ​​and reward mapping for a candidate neural network.

[0017] Figure 6 An example of a user interface presentation showing agent behavior.

[0018] Figure 7 Another example of a user interface presentation showing agent behavior.

[0019] Like reference numbers and designations in the various drawings refer to like elements. DETAILED DESCRIPTION

[0020] This specification describes a reinforcement learning system that controls an agent that interacts with an environment by processing, at each of a plurality of time steps, data representing the current state of the environment at that time step (i.e., an "observation") to select an action to be performed by the agent.

[0021] At each time step, the state of the environment at that time step depends on the state of the environment at the previous time step and the action performed by the agent at the previous time step.

[0022] In some embodiments, the environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle that navigates the environment. In particular, the environment can be a real-world environment in which multiple agents (i.e., multiple autonomous vehicles or multiple robots) are operating. In these cases, control strategies for controlling other agents are typically not available to the system.

[0023] In these embodiments, observations may include, for example, one or more of images, object position data, and sensor data to capture observations as the agent interacts with the environment, such as sensor data from images, range or position sensors, or from actuators.

[0024] For example, in the case of a robot, observations may include data characterizing the current state of the robot, such as one or more of joint positions, joint velocities, joint forces, torques or accelerations (e.g., gravity-compensated torque feedback), and the global or relative pose of an item held by the robot.

[0025] In the case of a robot or other mechanical agent or vehicle, observations may similarly include one or more of position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more components of the agent. Observations may be defined in one, two, or three dimensions, and may be absolute and / or relative.

[0026] Observations may also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or image or video data, for example from a camera or LIDAR sensor, e.g., data from sensors of the agent or from sensors placed separately from the agent in the environment.

[0027] In these embodiments, the actions can be control inputs for controlling a robot (e.g., torques of a robot's joints or higher-level control commands), or control inputs for controlling an autonomous or semi-autonomous land, air, or sea vehicle (e.g., torques of a vehicle's control surfaces or other control elements or higher-level control commands).

[0028] In other words, an action may include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or a component of another mechanical agent. The action data may additionally or alternatively include electronic control data (such as motor control data), or more generally, data for controlling one or more electronic devices within an environment, the control of which has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land or air or sea vehicle, an action may include actions for controlling the travel (e.g., steering) and movement (e.g., braking and / or acceleration) of the vehicle.

[0029] In some other applications, the agent can control actions in a real-world environment including various equipment, for example, actions in a data center, a power / water distribution system, or a manufacturing plant or service facility. The observations can then be related to the operation of the plant or facility. For example, the observations can include observations of power usage or water usage of equipment, or observations of power generation control or power distribution control, or observations of resource usage or waste generation. These actions can include actions to control or impose operating conditions of various pieces of equipment in the plant / facility, and / or actions that cause changes in settings in the operation of the plant / facility, for example, to adjust or turn on / off components of the plant / facility.

[0030] In the case of an electronic agent, observations may include data from one or more sensors monitoring a portion of a plant or service facility (such as current, voltage, power, temperature, and other sensors) and / or electronic signals representing the functionality of various pieces of electronic and / or mechanical equipment.

[0031] Figure 1 An example reinforcement learning system 100 is shown. Reinforcement learning system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0032] The system 100 controls an agent 102 interacting with an environment 104 by selecting actions 106 to be performed by the agent 102 and then causing the agent 102 to perform the selected actions 106. Such repeated performance of actions by the agent typically results in repeated transitions of the state of the environment to new states.

[0033] System 100 includes policy neural network 110 and training engine 116 , and maintains a set of model parameters 118 for policy neural network 110 .

[0034] At each of a plurality of time steps, policy neural network 110 is configured to process inputs including current observations 120 characterizing a current state of environment 104 according to model parameters 118 to generate action selection outputs 122 (“action selection policies”).

[0035] The system 100 uses the action selection output 122 to select an action 106 to be performed by the agent at the current time step. Several examples of using the action selection output 122 to select an action 106 to be performed by the agent 102 are described next.

[0036] In one example, the action selection output 122 can define a probability distribution of actions in a set of possible actions that can be performed by the agent. For example, the action selection output 122 can include a corresponding numeric probability value for each action in the set of possible actions that can be performed by the agent. As another example, the action selection output 122 can include parameters of a distribution of the set of possible actions, such as parameters of a multivariate normal distribution of the set of actions when the set of actions is represented as a continuous space. For example, the system 100 can select actions to be performed by the agent by sampling actions according to their probability values ​​or by selecting the action with the highest probability value.

[0037] In another example, the action selection output 122 may directly define the action to be performed by the agent, such as by defining torque values ​​that should be applied to the joints of the robotic agent.

[0038] In another example, the action selection output 122 may include a corresponding Q-value for each action in a set of possible actions that may be performed by the agent. The system 100 may process the Q-values ​​(e.g., using a softmax function) to generate a corresponding probability value for each possible action, which may be used to select an action to be performed by the agent (as described above). The system 100 may also select the action with the highest Q-value as the action to be performed by the agent.

[0039] The Q-value of an action is an estimate of the “reward” that would result from the agent performing the action in response to the current observation 120 and then selecting a future action to be performed by the agent 102 based on the current values ​​of the policy neural network parameters.

[0040] A return refers to a cumulative measure of the "rewards" 124 received by an agent, e.g., a time-discounted sum of the rewards. An agent may receive a corresponding reward 124 at each time step, where the reward 124 is specified by a scalar value and represents, e.g., the agent's progress in completing an assigned task.

[0041] In some cases, the system 100 may select an action to be performed by the agent according to an exploration strategy. For example, the exploration strategy may be an ∈-greedy exploration strategy, in which the system 100 selects an action to be performed by the agent according to the action selection output 122 with probability 1-∈, and randomly selects an action with probability ∈. In this example, ∈ is a scalar value between 0 and 1. As another example, the system may add randomly sampled noise to the action selection output 122 to generate a noise output, and then use the noise output instead of the output 122 to select an action.

[0042] The policy neural network 110 includes a temporal hierarchical recurrent neural network 112. The temporal hierarchical recurrent neural network 112 in turn includes two recurrent neural networks: a fast-update recurrent neural network (RNN) that updates its hidden state at each time step and a slow-update recurrent neural network that updates its hidden state at time steps less than all time steps.

[0043] In this specification, the hidden state of the fast-updated recurrent neural network will be referred to as the "fast-updated hidden state", and the hidden state of the slow-updated recurrent neural network will be referred to as the "slow-updated hidden state".

[0044] As will be described in more detail below, fast-updating RNNs are called "fast-updating" because the fast-updating hidden state is updated at every time step. In contrast, slow-updating RNNs are called "slow-updating" because the slow-updating hidden state is not updated at every time step, but is only updated when certain criteria are met.

[0045] At each time step, the fast-update recurrent neural network receives an input including the observation 120 and the slow-update hidden state of the slow-update recurrent neural network and uses the input to update the fast-update hidden state.

[0046] The policy neural network 110 then uses this updated fast-updated hidden state to generate an action selection output 122. By utilizing the temporal hierarchical RNN 112, the system 100 can select an action at each time step that is consistent with the agent's long-term plan, and the system 100 can effectively control the agent even on tasks that may require that the action selected by the user at any given time step depends on data received from observations at a large number of time steps prior to the given time step.

[0047] Reference below Figure 2-4 The operation and architecture of the policy neural network 110 and the temporal hierarchical recurrent neural network 112 are described in more detail.

[0048] Training engine 116 is configured to train policy neural network 110 by repeatedly updating model parameters 118 of policy neural network 110 based on the agent's interactions with the environment (ie, using observations 120 and rewards 124 received as a result of the agent's interactions with the environment).

[0049] The training engine 116 can use the data collected from the agent's interaction with the environment, using reinforcement learning techniques, to train the policy neural network 110 to increase the reward (i.e., the cumulative measure of the reward) received by the agent. Because the reward measures the agent's progress in completing a task, training the policy neural network 110 to increase the reward results in the agent being able to successfully complete the specified task while being controlled by the policy neural network 110.

[0050] However, for some tasks, external rewards 124 may not be sufficient to generate a high-quality learning signal that allows training engine 116 to train policy neural network 110 to have high performance on the task. For example, rewards 124 may be very sparse and very delayed. That is, even if a given action performed at a given time step contributes to the successful completion of the task, the corresponding reward may not be received for many time steps after the given time step.

[0051] As a special example, some tasks may require a large number of operations to be successfully completed. However, reward 124 may be non-zero only when the task reaches a terminal state, e.g., may be positive only when the environment reaches a final state where the task is complete. This may make it difficult to train policy neural network 110 using only sparse reward 124 as a learning signal.

[0052] In these cases, the system 100 can utilize the additional data 152 that can be extracted from the environment to generate an improved reward signal even when data specifying how the extracted additional data 152 is related to task performance is not specified before training begins. In particular, the reward mapping engine 160 can map the additional data 152 (and optionally the reward 124 in the non-zero case) to a learned reward value 162 at each time step, and the learned reward 162 can be used by the training engine 116 to update the model parameters 118 (i.e., rather than relying solely on the sparse reward 124). Figure 2 and Figure 5 Describe learning rewards in more detail.

[0053] In addition, the system 100 can generate data identifying time-extended behavior patterns of the agent interacting with the environment and provide it to the user for presentation. That is, the system 100 can generate user interface data and provide the user interface data to the user for presentation on the user device. This can facilitate the evaluation of the performance of the agent by allowing the user to analyze the behavior of the agent while being controlled by the policy neural network. Figure 6 and Figure 7 Describes the generation of such user interface data.

[0054] Figure 2 shows the operation of the reinforcement learning system at time step t during the agent's interaction with the environment.

[0055] exist Figure 2 In the example of , the observation 120 received at time step t is, for example, an image of the environment captured by a camera sensor of the agent at time step t. However, as described above, in addition to or instead of images, the observation may also include other data characterizing the state of the environment.

[0056] Figure 2 The example of shows how the fast-updated hidden state 210 and the slow-updated hidden state 220 are updated over a sequence of time steps including time step t. Figure 2 As can be seen in the example of , the fast-update RNN updates the fast-update hidden state at each time step using the input that includes the observation at that time step. On the other hand, the slow-update RNN does not update the slow-update hidden state at each time step. Figure 2 In the example, the time steps where the hidden state is updated are those where the arrow representing the hidden state is connected to the black circle, and the time steps where the hidden state is not updated are those where there is no black circle. For example, at the time steps before time step t in the sequence, there is no update to the state of the slow-updating RNN.

[0057] At any given time step, if the criteria for updating the slow-updated hidden state are met, the system uses the slow-updated RNN to process the slow-updated input including the fast-updated hidden state (i.e., the final fast-updated hidden state after the time step before the given time step) to update the slow-updated hidden state. Therefore, at time step t, the input of the slow-updated RNN can be the fast-updated hidden state after time step t-1, and the slow-updated RNN can use this input to update the slow-updated hidden state at time step t.

[0058] The slow-updating RNN can have any suitable recurrent architecture. For example, the slow-updating RNN can be a stack of one or more long short-term memory (LSTM) layers. When there is more than one layer in the stack, the slow-updating hidden state can be the hidden state of the last LSTM layer in the stack.

[0059] When the criteria is not met, the system avoids updating the slowly-updated hidden state at that time step, i.e., the slowly-updated hidden state is not updated before it is used for further processing at that time step.

[0060] At each time step, the system generates parameters of a prior distribution 222 of possible values ​​of the latent variable based on the slowly updating hidden state, such as by applying a linear transformation to the slowly updating hidden state. Typically, the latent variable is a vector with a fixed dimension, so the prior distribution is a multivariate distribution of possible values ​​of the latent variable. For example, the prior distribution can be a normal distribution, so the parameters can be the mean and covariance of the multivariate normal distribution. When the slowly updating hidden state is not updated at a given time step, the system can reuse the most recently calculated prior parameters, rather than having to recalculate the prior parameters based on the same hidden state as in the previous time step.

[0061] The system then processes the fast-update input using the fast-update RNN to update the fast-update hidden state, where the fast-update input includes the observation 120, the slow-update hidden state, and the parameters 222 of the prior distribution.

[0062] Like the slow-update RNN, the fast-update RNN can have any suitable recurrent architecture. For example, the fast-update RNN can be a stack of one or more long short-term memory (LSTM) layers. When there is more than one layer in the stack, the fast-update hidden state can be the hidden state of the last LSTM layer in the stack.

[0063] When the observation is an image (like in Figure 2When the fast-update RNN is used for training (as in the example of ), or other high-dimensional data, the fast-update RNN may include a convolutional neural network (CNN) or other encoder neural network that encodes the observations before processing the encoded representations of the observations through the recurrent layers of the fast-update RNN.

[0064] In addition, in some embodiments, the fast-updating RNN and the slow-updating RNN are enhanced with shared external memory, i.e., both RNNs read and write from the same shared external memory as part of updating the corresponding hidden states. An example architecture for enhancing RNNs with external memory that can be used by the system is the Differentiable Neural Computer (DNC) memory architecture, which is described in more detail in the following literature: Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, AgnieszkaGrabska-Barwinska, Sergio Gomez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626): 471, 2016.

[0065] Once the fast-updated hidden state has been updated at time step t, the system uses the fast-updated hidden state to generate parameters of the posterior distribution 212 of possible values ​​of the latent variable, such as by applying a linear transformation to the slow-updated hidden state.

[0066] The system then samples the latent variable 232 from the posterior distribution 212 , i.e., samples a possible value of the latent variable according to the probability in the posterior distribution 212 .

[0067] The system then uses the sampled latent variables 232 to select an action 106 to perform in response to the observation 120 .

[0068] In particular, the system processes the sampled latent variables using one or more policy neural network layers to generate a policy output (ie, action selection output 122 ), and then uses the policy output to select an action 106 .

[0069] The fast-update input processed by the fast-update RNN at any given time step may also include additional inputs (besides observations, slow-update hidden states, and prior parameters). For example, the fast-update hidden state at time step t is Can satisfy:

[0070]

[0071] Among them, u t is the observation x at time step t t The encoding representation of a t-1 is the action performed at the previous time step, r t-1 is the reward from the previous time step (external or learned), is the slowly updated hidden state at time t, is the previous fast update hidden state, and are the mean and covariance of the prior distribution generated using the slow update hidden state, z t-1 is the latent variable sampled at the previous time step.

[0072] By sampling latent variables from a distribution rather than directly using fast-updated hidden states to select actions, the system can effectively account for uncertainty in situations where the received observations do not fully characterize the true state of the environment, i.e., when the true state is unclear even given all observations received by the agent so far.

[0073] When time step t occurs during learning (or when time step t occurs after learning, but the reward is part of the input to the fast-update RNN) and the system is using the learned reward to augment the external reward, the system also receives additional data 152 extracted from the environment and applies the current reward mapping to the additional data 152 to generate a learned (or "internal") reward 162. This learned reward can then be used to update the policy neural network 110. Applying the mapping and updating the mapping data will be described in more detail below.

[0074] In particular, the system may train the neural network 110 to maximize the rewards received using reinforcement learning.

[0075] As a specific example, in some cases, the neural network 110 also generates a value output (hereinafter also referred to as a "baseline output") based on the sampled latent variables, which is an estimate of the reward generated from the environment in the current state. In these cases, the system can use actor-critic reinforcement learning techniques to train the neural network 110.

[0076] During training, the system uses a prior distribution generated by the prior parameters to regularize the latent variables. This helps ensure that the slow-update RNN and the fast-update RNN capture long-term temporal correlations and promote memory of information received in observations. In particular, during training, the system enhances reinforcement learning techniques by training the fast-update recurrent neural network and the slow-update recurrent neural network to minimize the divergence (e.g., KL divergence) between the prior distribution and the posterior distribution. In some cases, to prevent the two distributions from matching, the system also includes a term in the loss function during training that penalizes the KL divergence between the prior distribution and a multivariate Gaussian distribution with a mean of 0 and a standard deviation of 0.1 or some other small fixed value.

[0077] As will be described in more detail below, during training, the system can also use the hidden states of the fast-update recurrent neural network to generate corresponding auxiliary outputs for each of one or more auxiliary tasks, and train the neural network on the auxiliary tasks based on the auxiliary outputs of the auxiliary tasks.

[0078] Figure 3A and Figure 3B is a diagram illustrating an example architecture of various components of policy neural network 110. Figure 3A The legend shown, identifying the various neural network components represented by the symbols in the diagram, also applies to Figure 3B Figure in .

[0079] Figure 3A An example of a high-level architecture 310 of the policy neural network 110 is shown. In particular, the architecture shows a visual embedding (convolutional) neural network 312 that processes input observations to generate an encoded representation, and a recurrent processing block 314 that includes a slow-update RNN and a fast-update RNN that generate latent variables.

[0080] The architecture 310 also shows: a policy neural network layer 322 that generates an action selection output conditioned on the latent variables; a baseline neural network layer 320 that uses the latent variables to generate a baseline score for use in training the policy neural network when using a reinforcement learning technique that requires a baseline state value score, such as an actor-critic based technique, such as the Importance Weighted Actor-Learner (IMPALA) technique described in the following literature: IMPALA: Scalable Distributed Deep-RL with ImportanceWeighted Actor-Learner Architectures, Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, Koray Kavukcuoglu. As described above, the system can enhance such training by adding additional loss terms that regularize the latent variables using a prior distribution.

[0081] The architecture 310 also includes a number of auxiliary components that can be used to improve the training of the policy neural network, such as a reward prediction neural network 316 that attempts to predict the next reward to be received, and a pixel control neural network 318 that predicts how much the currently observed pixel will change in the next observation. These neural networks can be trained jointly with the rest of the policy neural network, and gradients can be back-propagated from these neural networks into the policy neural network to improve the representations learned by the policy neural network during training.

[0082] Figure 3A Also shown is the detailed architecture 330 of a visual embedding neural network that receives observations and generates an encoded representation that is provided as part of the input to the loop processing block 314.

[0083] Figure 3B A detailed architecture of a policy neural network 340 , a detailed architecture of a baseline neural network 350 , and a detailed architecture of a loop processing block 360 , which includes a variational unit (VU) described in more detail in a detailed VU architecture 370 , are shown.

[0084] Figure 4is a flow chart of an example process 400 for selecting an action to be performed by an agent. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed reinforcement learning system (e.g., Figure 1 The reinforcement learning system 100) can perform process 400.

[0085] The system determines whether the criteria for updating the slow-update hidden state are met (step 402). As described above, the criteria are typically met at less than all time steps.

[0086] For example, the criterion may be satisfied only at predetermined intervals during the agent's interaction with the environment. That is, the criterion may be satisfied at every N time steps, where N is an integer greater than 1. For example, N may be an integer in the range of 5 to 20, inclusive.

[0087] As another example, the system may determine at each time step a difference metric between the observation at that time step and the observation at the previous time step at which the hidden state of the slow-updating recurrent neural network was updated. In this example, the criterion is met at a given time step only if the difference metric meets a threshold, i.e., when the observation is sufficiently different from the observation at the previous time step at which the hidden state was updated. For example, the distance metric may be the cosine similarity between the encoded representations generated by the encoder neural network, and the criterion may be met only if the cosine similarity is below a threshold similarity.

[0088] When the criteria are met, the system updates the slowly updating hidden state as described above (step 404). When the criteria are not met, the system refrains from updating the slowly updating hidden state.

[0089] The system receives an observation at the time step (step 406).

[0090] The system processes the fast update input as described above to update the fast update hidden state (step 408).

[0091] The system uses the fast-updated hidden state to select an action (step 410). For example, the system can use the fast-updated hidden state to sample latent variables as described above, and then use one or more policy neural network layers to process the sampled latent variables to generate a policy output.

[0092] As described above, in some embodiments, the system enhances the training of the neural network by using learned reward values ​​instead of or in addition to actual external reward values ​​received from the environment.

[0093] In particular, the system uses a reward map that maps data extracted from the environment to a reward value. Typically, the extracted data includes a corresponding numerical value for each of one or more reward signals that can be directly observed from the environment (e.g., directly measured by the agent's sensors), can be determined based on the agent's sensor measurements, or can be determined based on other information about the environment and the agent accessible to the system. As a specific example, one or more reward signals can identify the location of the agent in the environment. As another specific example, one or more reward signals can identify the distance of the agent relative to a known object or location in the environment. As yet another example, one or more reward signals can be external rewards received from the environment, such as a reward indicating that a task has been successfully completed or unsuccessfully terminated.

[0094] Assuming that the reward signal can be mapped to a reward value, the reward mapping data identifies, for each reward signal, a partition of the possible values ​​of the reward signal into a plurality of partitions, and then maps each partition to a corresponding reward value of the reward signal. Thus, for a given reward signal, the data maps different partitions to different reward values.

[0095] When more than one reward signal is extracted from the environment at a given time step, the system can combine the reward values ​​of the reward signals (e.g., average or sum) to generate a final reward value.

[0096] In embodiments where the system learns the reward mapping, the system may maintain multiple candidate neural networks during training, and may use the set of multiple candidates to jointly update parameter values ​​and learn the reward mapping.

[0097] In particular, during training, the system may maintain, for each of the candidate neural networks, data identifying parameter values ​​of the candidate neural network and data identifying a reward mapping used during training of the candidate neural network.

[0098] During training, for each candidate neural network, the system performs the following references in parallel: Figure 5 The training operation described above is used to repeatedly update the parameter values ​​and reward map used by the candidate neural network.

[0099] Once training is complete, i.e., after repeatedly performing the training operation, the system selects trained network parameter values ​​from the parameter values ​​in the maintenance data of the candidate neural network. For example, after repeatedly performing the training operation, the system can select the maintained parameter values ​​of the candidate neural network with the best performance on the reinforcement learning task. Thus, the system can continuously improve the reward mapping used by the candidate neural network in order to enhance the training process and generate a higher quality trained neural network than can be achieved using only external rewards.

[0100] Figure 5 is a flow chart of an example process 500 for updating parameter values ​​and reward mappings of a candidate neural network. For convenience, process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed reinforcement learning system (e.g., Figure 1 The reinforcement learning system 100) can perform process 500.

[0101] The system can repeat the process 500 in parallel for all candidate neural networks in the set. In some cases, the system performs the process 500 asynchronously for all candidate neural networks.

[0102] The system trains the candidate neural network using the maintained reward map of the candidate neural network (step 502). That is, the system trains the candidate neural network using reinforcement learning techniques (e.g., the techniques described above) on the agent's interaction with the environment to update parameter values ​​of the candidate neural network. During training, the system uses data extracted from the environment and the reward map of the candidate neural network to generate rewards used in training.

[0103] For example, the system may train the candidate neural network for a predetermined number of time steps or for a predetermined number of task episodes or for a predetermined amount of time to complete step 502 .

[0104] The system determines a quality metric for the trained candidate neural network, which represents how well the candidate neural network is able to control the agent to perform the task (relative to other candidate neural networks) (step 504). An example of a quality metric may be the number of task segments that are successfully completed when the agent is controlled by the trained candidate neural network. In some cases, rather than using an external reward as part of the reward mapping, the system may use the average external reward received when the candidate neural network is used to control the agent as the quality metric.

[0105] The system uses the quality metric to determine new network parameter values ​​and a new reward mapping for the candidate neural network (step 506).

[0106] Typically, the system can update new network parameter values, a new reward mapping, and optionally other hyperparameters of training (e.g., the interval N as described above) so that the weaker neural network copies the parameters of the stronger neural network while exploring the space of possible reward mappings and optionally other hyperparameters.

[0107] For example, the system may sample another candidate from the set and compare the quality metrics of the two candidates.

[0108] If a quality metric of one candidate is found to be worse than another candidate by more than a threshold amount, the system can copy the parameters, reward mappings, and hyperparameters of the better candidate for the worst performing candidate. The system can then explore new reward mappings and hyperparameters for the worst performing candidate. For example, the system can determine for each reward signal partition whether to modify the reward value to which the partition is mapped with a predetermined probability, and in response to determining to modify the reward value to which the partition is mapped, change the value to which the partition is mapped, for example, randomly perturbing the value in either direction by a fixed fraction of the current value

[0109] As another example, the system can sample one or more other candidate neural networks and determine whether the quality metric indicates that the candidate neural network performs better than all of the one or more other candidate neural networks.

[0110] In response to determining that the candidate neural network does not perform better than all one or more other candidate neural networks, the system (1) sets new network parameter values ​​to the maintained network parameter values ​​of the best performing sampled other candidate neural networks, and (2) sets the new reward mapping to a modified version of the maintained reward mapping of the best performing other candidate neural networks. For example, the system can generate the modified version by determining, for each reward signal partition, whether to modify the reward value to which the partition is mapped with a predetermined probability and, in response to determining to modify the reward value to which the partition is mapped, changing the value to which the partition is mapped (e.g., randomly perturbing the value in either direction by a fixed fraction of the current value).

[0111] In response to determining that the candidate neural network does perform better than all one or more other candidate neural networks, the system sets the new network parameter values ​​as the updated network parameter values ​​after training in step 502 and does not modify the reward mapping.

[0112] By repeatedly performing process 500 for all candidate neural networks, the system trains the neural networks while exploring the space of possible reward mappings to discover reward mappings that result in better agent performance on the task.

[0113] As described above, in some embodiments, the system can provide certain data derived from the agent's interaction with the environment for presentation to the user. Thus, the system can automatically identify time-extended behavioral patterns of the agent interacting with the environment and provide data identifying these patterns as output to the user device.

[0114] In some implementations, to discover these patterns, the system may allow the agent to interact with the environment and then determine multiple time series of features that characterize the state of the environment as the agent interacts with the environment.

[0115] The system can then train a variational autoencoder, in particular a variational sequence autoencoder, with the captured time series. The variational sequence autoencoder can include a recurrent neural network encoder coupled to a recurrent neural network decoder, the recurrent neural network encoder for encoding an input data sequence into a set of latent variables, the recurrent neural network decoder for decoding the set of latent variables to produce an output data sequence, i.e., to reproduce the output data sequence from the input data sequence. During training, the latent variables are constrained to be approximately defined distributions, such as Gaussian distributions.

[0116] In some embodiments, as the agent interacts with the environment, the system also captures a further sequence of observations of the environment. The system can then process the further sequence of observations using the trained variational sequence autoencoder to determine a sequence of the set of latent variables. Optionally, the system can process the sequence of the set of latent variables to identify a sequence of clusters in the space of latent variables, each cluster representing a time-extended behavior pattern of the agent.

[0117] The system can automatically identify and present to the user data identifying complex high-level behaviors of the agent. In an embodiment, the method represents these behaviors in a human-understandable manner, as further described below. For example, the clusters identified by the system can correspond to high-level behavior patterns that are more easily recognized by humans than the individual actions that make up the high-level behavior patterns.

[0118] Features characterizing the state of the environment used to identify behavior may be agent-centric features, i.e., features related to the high-level behavior performed by the agent and represented by the clusters. Thus, these features may include, for example, features characterizing the position or state of the agent relative to the environment. For example, they may be features defined relative to the agent, such as the position or orientation of the agent or a part of the agent, or properties of the agent, or states of the agent or the environment that are related to the short-term or long-term reward of the agent.

[0119] In some embodiments, features may be manually defined and provided as input to the method. These features may be high-level features, in the sense that features may be defined as macroscopic properties of the agent-environment system, and in particular may be interpretable by a human in the context of a task at a high enough level. For example, features may be related at a high level to the definition or achievement of a goal, and / or features may be expected to be meaningful components of some behavioral patterns. Features may be captured directly from the environment through observations separate from observations of the agent, e.g., where properties of the environment are directly available because they are in a simulated environment, and / or they may be derived from observations of the agent, and / or they may be learned.

[0120] The variational sequence autoencoder processes feature sequences on a time scale suitable for the desired behavior pattern. For example, if it is desired to characterize behavior on a time scale of one to several seconds, the captured time series used to train the variational sequence autoencoder can have a similar time scale. The variational sequence autoencoder learns to encode or compress the feature sequence into a set of latent variables representing the sequence. The latent variables can be constrained to have a Gaussian distribution; they can include mean and standard deviation vectors. The recurrent neural network encoder and decoder can each include an LSTM (Long Short-Term Memory) neural network.

[0121] Thus, the set of latent variables of the variational sequence autoencoder can provide a latent representation of the observation sequence based on the high-level features. A mixture model can be used to process the sequence of latent variable sets derived from further observations to identify clusters. For example, each latent variable set can be mapped to one of K components or clusters using a mixture model, which can be a Gaussian mixture model. Each component or cluster can correspond to a different time-extended behavior derived from features representing the high-level characteristics of the agent-environment system. These can be considered to be prototypical behaviors of the agent, each of which extends over the time range of the sequence used to train the autoencoder. Therefore, processing further observation sequences ultimately leads to determining which prototypical behavior in the set of prototypical behaviors the agent engages in at each time in a series of times.

[0122] Optionally, these behaviors (more specifically, clusters) can be represented, for example graphically, in conjunction with corresponding representations of the agents and / or environments of some or all of the clusters. The representation of the clusters can be, for example, a chart with time on one axis and identified behavioral patterns on the same axis or another axis. The representation of the agents and / or environments can be in any convenient form, such as an image and / or a collection of features. There may be an association between clusters and corresponding states, such as representing these clusters and corresponding states in conjunction with each other, or hovering over a representation of a cluster with a mouse to provide an indication of the state.

[0123] Figure 6 An example of a user interface presentation 600 of agent behaviors is shown. In particular, user interface presentation 600 shows behaviors that have been discovered using the techniques described above. In particular, presentation 600 shows clusters of behaviors along axis 602 and the number of steps during a period in which the behaviors were engaged along axis 604. Thus, the height of a bar in presentation 600 representing a given behavior represents the frequency with which the agent performed that behavior during that period. The system can allow a user to access more information about a behavior, such as a representation of the state of the environment when the agent engaged in the behavior, by hovering over, selecting, or otherwise interacting with a bar in presentation 600.

[0124] Figure 7 Another example of a user interface presentation 700 of agent behaviors is shown. In particular, user interface presentation 700 is a time graph that shows how often particular clusters of behaviors are performed as the period of a task progresses, i.e., shows when, in terms of time, certain clusters of behaviors are performed within a period ("period time"). Presentation 700 includes a corresponding row for each of 32 different clusters of behaviors, with white bars representing time periods during the period when the agent was performing one or more behaviors in the cluster. For example, white bar 702 shows that the agent was performing behaviors in cluster 12 at a particular time period during the period, white bar 704 shows that the agent was performing behaviors in cluster 14 at a different later time period during the period, and white bar 706 shows that the agent was performing behaviors in cluster 32 at another, even later time period during the period.

[0125] As with representation 600, the system may allow a user to access more information about a behavior, such as a representation of the state of the environment when the agent was engaging in the behavior, by mousing over, selecting, or otherwise interacting with a bar in presentation 700.

[0126] Although the subject technology has been described primarily in the context of a physical real-world environment, it will be appreciated that the technology described herein can also be utilized in non-real-world environments. For example, in some embodiments, the environment can be a simulated environment, and the agent can be implemented as one or more computers that interact with the simulated environment.

[0127] The simulated environment may be a motion simulation environment (eg, a driving simulation or a flight simulation), and the agent may be a simulated vehicle traveling through the motion simulation. In these embodiments, the action may be a control input for controlling a simulated user or a simulated vehicle.

[0128] In another example, the simulated environment may be a video game, and the agent may be a simulated user playing the video game. Typically, in the case of a simulated environment, observations may include simulated versions of one or more previously described observations or observation types, and actions may include simulated versions of one or more previously described actions or action types.

[0129] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers, being configured to perform a particular operation or action means that the system has installed on the system software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs, being configured to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0130] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible non-temporary storage medium, which are used to be executed by a data processing device or control the operation of the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0131] The term "data processing apparatus" refers to data processing hardware and covers all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, or a plurality of processors or computers. The apparatus may also be or also include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0132] A computer program, which may also be referred to or described as a program, software, software application, applet, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages; and the computer program may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0133] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and it may be stored on storage devices in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed differently.

[0134] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0135] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits (e.g., FPGAs or ASICs), or by a combination of special purpose logic circuits and one or more programmed computers.

[0136] A computer suitable for executing a computer program can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or a central processing unit of any other kind. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by or incorporated into a special logic circuit. Typically, a computer will also include or be operably coupled to one or more large-capacity storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, to receive data from them or to transmit data to them, or both. However, a computer does not need such a device. In addition, a computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few examples.

[0137] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; CD ROM and DVD-ROM disks.

[0138] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. Moreover, the computer may interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application), and receiving a response message from the user in return.

[0139] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for handling both the ordinary and computationally intensive parts of the machine learning training or production, i.e., the inference, workload.

[0140] The machine learning model can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework).

[0141] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computing system having a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification), or includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0142] A computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML web page) to a user device, for example, in order to display data to a user interacting with a device acting as a client and to receive user input from the user. Data generated at the user device (e.g., the result of the user interaction) can be received from the device at the server.

[0143] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or on the scope of the claimed protection, but rather as descriptions of specific features of specific embodiments of specific inventions. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations, and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0144] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order or that all of the operations shown be performed to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0145] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes described in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for reinforcement learning to select an action to be performed by an agent interacting with an environment, the method comprising at each time step of a plurality of time steps: determining whether a criterion for updating a hidden state of a slow-update recurrent neural network is met at the time step; When the criteria are met, processing the slow-update input including the hidden state of the fast-update recurrent neural network to update the hidden state of the slow-update recurrent neural network; receiving an observation characterizing a current state of the environment at the time step; processing a fast-update input for the time step using a fast-update recurrent neural network to update a hidden state of the fast-update recurrent neural network, the fast-update input comprising (i) the observation and (ii) the hidden state of the slow-update recurrent neural network; as well as After updating the hidden state of the fast-update recurrent neural network: using a hidden state of a fast recurrent neural network to select an action to be performed in response to the observation; wherein the observation is data or a signal characterizing a state of an environment; and the agent is a robot or one of autonomous or semi-autonomous land, air or sea vehicles navigating in the environment, or the agent is an agent for controlling equipment in a data center, a power / water distribution system, or a manufacturing plant or service facility; wherein the criterion is satisfied at less than all of the plurality of time steps.

2. The method of claim 1 , further comprising, at each time step in a plurality of time steps: When the criterion is not met, updating the hidden state of the slow-update recurrent neural network is avoided before the hidden state is used as part of the fast-update input for the time step.

3. The method according to any one of claims 1 to 2, wherein: Every N time steps meets the criterion, and where N is a fixed integer greater than 1.

4. The method according to any one of claims 1-2, further comprising at each time step in the plurality of time steps: Determine a difference measure between the observation at the time step and the observation at the previous time step at which the hidden state of the slow-update recurrent neural network was updated, where The criterion is satisfied at the time step only if the difference measure satisfies a threshold.

5. The method according to any one of claims 1 to 2, wherein: Using a fast-updating recurrent neural network hidden state to select the action includes: Generate posterior parameters of the posterior distribution of possible latent variables based on the hidden states of the fast-updated recurrent neural network; sampling latent variables from a posterior distribution parameterized by the posterior parameters; and The action is selected using the sampled latent variable.

6. The method according to claim 5, wherein: Selecting the actions includes: Use a policy neural network to process the sampled latent variables to generate policy outputs, and The action is selected using the policy output.

7. The method according to claim 5, wherein: The fast update input also includes the latent variable sampled at the previous time step.

8. The method according to claim 5, further comprising: At each time step in the plurality of time steps, prior parameters of a prior distribution of possible latent variables are generated based on hidden states of the slowly-updated recurrent neural network.

9. The method according to claim 8, wherein: The fast update input also includes prior parameters of the prior distribution generated at the previous time step.

10. The method according to any one of claims 1-2 and 6-9, further comprising: Receiving a reward in response to the agent performing the selected action; as well as A fast-update recurrent neural network and a slow-update recurrent neural network are trained based on the selected actions and the rewards using reinforcement learning techniques.

11. The method according to any one of claims 7-8, further comprising: Receiving a reward in response to the agent performing the selected action; as well as training a fast-update recurrent neural network and a slow-update recurrent neural network based on the selected action and the reward using reinforcement learning techniques; Among them, prior distributions are used to regularize latent variables during the training of fast-update recurrent neural networks and slow-update recurrent neural networks.

12. The method according to claim 11, wherein: Using prior distributions to regularize latent variables during training of fast-update RNNs and slow-update RNNs includes: Train a fast-update recurrent neural network and a slow-update recurrent neural network to minimize the divergence between the prior and posterior distributions.

13. The method of claim 10 further comprising, at each of the plurality of time steps, using a hidden state of a fast-updating recurrent neural network to generate a value output that is an estimate of a reward generated from the environment in the current state, and wherein, The training includes training a fast-update recurrent neural network and a slow-update recurrent neural network based on the reward, the selected action, and the value output using an actor-critic technique.

14. The method according to any one of claims 10, further comprising: generating a value output using a hidden state of a fast-update recurrent neural network at each of the plurality of time steps, the value output being an estimate of a reward resulting from the environment in the current state, and wherein the training comprises training the fast-update recurrent neural network and the slow-update recurrent neural network based on the rewards, the selected actions, and the value outputs using an actor-critic technique; Wherein, generating the value output includes using sampled latent variables to generate the value output.

15. The method of claim 10, further comprising at each time step in the plurality of time steps: generating a corresponding auxiliary output for each of the one or more auxiliary tasks using a fast-updating hidden state of the recurrent neural network; and For each of the one or more auxiliary tasks, a fast-update recurrent neural network and a slow-update recurrent neural network are trained on the auxiliary task based on an auxiliary output of the auxiliary task.

16. The method according to claim 15, wherein: Generating a corresponding auxiliary output for each of the one or more auxiliary tasks includes generating the corresponding auxiliary output using the sampled latent variables.

17. The method according to any one of claims 1-2, 6-9, 12, 13 and 15-16, wherein The slow-update recurrent neural network and the fast-update recurrent neural network share an external memory.

18. The method of any one of claims 1-2, 6-9, 12, 13 and 15-16, further comprising at each time step in the plurality of time steps: Causes the agent to perform the selected action.

19. The method according to claim 10, wherein: The rewards include: After executing the selected action, obtain the data extracted from the environment, and The obtained data is mapped to the rewards using a reward map.

20. A system comprising one or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform the method according to any preceding claim.

21. One or more computer storage media storing instructions which, when executed by one or more computers, cause the one or more computers to perform the method according to any preceding claim.

Images