Using hindsight modeling to train action selection neural networks

Abstract

A reinforcement learning method and system for selecting actions to be performed by a reinforcement learning agent interacting with an environment. A causal model is implemented by a hindsight model neural network and trained using hindsight, i.e., using future trajectories of environmental states. Because the method and system do not have access to this future information when selecting actions, the hindsight model neural network is used to train a model neural network that is conditioned on data from current observations and learns to predict the output of the hindsight model neural network.

Description

Background Art

[0001] This specification relates to reinforcement learning.

[0002] In a reinforcement learning system, an agent interacts with an environment by performing actions, which are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.

[0003] Based on the output of a neural network, some reinforcement learning systems select an action to be performed by the agent in response to receiving a given observation.

[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output given a given input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer serves as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from its input based on the current values ​​of its corresponding parameter set. Summary of the Invention

[0005] This specification generally describes a reinforcement learning method and system for selecting actions to be performed by a reinforcement learning agent interacting with an environment. Some embodiments of the described method and system are directed to learning causal models using hindsight. This guides action selection and, in particular, focuses the model on aspects of the environment that are useful for estimating the current state value of the environment. Throughout this specification, references to state values ​​may include state values ​​in conjunction with actions.

[0006] In one aspect, a method of reinforcement learning is described. The method may include training an action selection neural network system to select an action to be performed by an agent performing a task in an environment. The action selection neural network system may be configured to receive data from observations representing a current state of the environment. The action selection neural network system may also be configured to receive data from an output of a model neural network. The action selection neural network system may process the input data according to action selection neural network system parameters to generate an action selection output for selecting an action to be performed by the agent.

[0007] The model neural network can be configured to receive input from observations (obtained from) representing the current state of the environment. The output of the model neural network can represent a predicted state trajectory, including a series of k predicted future states of the environment starting from the current state.

[0008] The method may include training a hindsight model neural network having an output representing a state trajectory, the state trajectory comprising a series of k states of an environment starting from a state of the environment at time step t. The training may include processing data from one or more observations (e.g., a series of observations) representing the state of the environment at time step t and a series of k subsequent time steps, either online or offline, and adjusting parameters of the hindsight model neural network using a training target at time step t. The method may also include training the output of the model neural network to approximate the output of the hindsight model neural network.

[0009] In an embodiment, the hindsight model can learn from a richer information stream than that provided by rewards from the environment, which are typically used in model-free training of reinforcement learning systems. For example, the hindsight model can learn to represent trajectories of states that, when appropriate, resulted in rewards.

[0010] More specifically, the hindsight model can learn to represent specific aspects of the state trajectory that are important for receiving rewards. Thus, rather than learning to model all the details in the observation, the hindsight model has a low-dimensional feature vector representation output, and optionally a relatively short look-ahead (i.e., hindsight) time window. These encourage the hindsight model to focus on modeling specific aspects of the observation that are useful for the training goal (e.g., state value or state-action value predictions), thereby speeding up training and reducing necessary computation. However, since this information is only available in hindsight, the model neural network is trained to predict this from the current state of the environment using the hindsight model. The hindsight model can process a fixed number k of future states; in embodiments, k can be less than 50, 20, or 10, such as about 5.

[0011] In some embodiments, the hindsight model can be trained to model aspects of the observations that can help provide better predictions of the values ​​of states (and state-action pairs) and thus be useful in training the action selection neural network. In this way, the model can be trained to focus on those aspects of the environment that are most important for the task to be performed. The model may also be simpler than otherwise.

[0012] In some embodiments, the training objective at time step t can be any objective relevant to the task to be performed.

[0013] For example, the training objective at time step t may include a state-value objective at time step t, or may be derived from one or more state-action-value objectives at time step t. For example, the training objective at time step t may define the expected return from the state of the environment at time step t, such as the expected cumulative reward received by the agent from the state of the environment at time step t.

[0014] More specifically, training a hindsight model can include processing the output of a hindsight model neural network and an observation representing the state of the environment at time step t using a hindsight value neural network to generate an estimated hindsight value or state-action value for the state of the environment at time step t. The method can then backpropagate the gradient of an objective function that depends on the difference between the estimated hindsight value or state-action value for the state of the environment at time step t and the training target at time step t to update parameters of the hindsight value neural network and parameters of the hindsight model neural network. Thus, the process of learning hindsight values ​​can be used to train a hindsight model.

[0015] In some embodiments, the training target (G) at time step t includes the actual reward at time step t (e.g., when using Monte Carlo learning); in other embodiments, the training target at time step t includes the estimated reward at time step t (e.g., when using temporal difference TD learning). Here, the reward refers to the reward that is a cumulative measure of the rewards received by the agent, such as the time-discounted sum of the rewards. For example, the reward at time step t may include a combination of the actual reward at time step t+1 and the estimated rewards at subsequent time steps, such as the hindsight value of the state of the environment at time step t+1 reduced by a discount factor γ < 1 (TD(0)); or, for example, n-step rewards may be used, such as TD(λ)). The difference used to train the neural network may include the TD error difference.

[0016] Training the output of the model neural network to approximate the output of the hindsight model neural network can include backpropagating the gradient of an objective function that depends on the difference between the (vector-valued) outputs (i.e., between the features of the state trajectory and the features of the predicted state trajectory). These features can represent various aspects of the trajectory that can be used to predict the value of the state-action value at the current time step t. The difference can include, for example, the L2 norm or cross-entropy loss. The model neural network and the hindsight model neural network can be trained jointly or separately, for example, sequentially.

[0017] In some embodiments, the method, the hindsight model neural network is configured to learn at a slower rate than the action selection neural network system (e.g., by selecting a learning rate parameter) to encourage the output of the model neural network to learn to approximate the output of the hindsight model neural network.

[0018] In an embodiment, the output of the hindsight model neural network that represents the state trajectory and the output of the model neural network that represents the predicted state trajectory each include a (multi-dimensional) dimensionality-reduced representation of one or more observations of the environment. This can facilitate the acceleration or reduction of computational learning of useful representations of the (predicted) state trajectory and can help encourage representations that are useful for predicting state values. For example, the output of the hindsight model neural network that represents the state trajectory and the output of the model neural network that represents the predicted state trajectory each can include a feature vector with a dimension less than the dimension of their input or less than 20, 10, or 5 (but not a scalar).

[0019] The training model neural network and the hindsight model neural network can be performed online or offline (e.g., offline). In the case of online training, a k-state buffer can be provided so that the training can effectively provide the past k states.

[0020] Thus, the method may further include maintaining a memory that stores data representing trajectories generated as a result of the agent's interaction with the environment, each trajectory comprising data at each time step in a series of time steps, the data at each time step identifying at least an observation representing a state of the environment and a series of subsequent observations representing subsequent states of the environment for use in training the hindsight model neural network.

[0021] For offline training of an action-selecting neural network system, the replay buffer can store tuples, such as a sequence of tuples, including: an observation representing a state of the environment, an action performed by the agent in response to the observation, a reward received in response to the agent performing the action, and an observation representing a next state of the environment.

[0022] In general, training an action selection neural network system can include backpropagating the gradient of an objective function that depends on the difference between a state-value or state-action value of a current state of the environment determined (estimated) using a state-value neural network and an estimated reward or state-action value of the current state of the environment.

[0023] The action selection neural network system can be trained using any reinforcement learning method, including, for example, SARSA, Q-learning, and actor-critic techniques including advantage actor-critic techniques (where learned estimates of state values ​​are used as baselines). The output of the action selection neural network system can directly indicate actions, such as a continuous series of actions, or can parameterize a distribution from which actions are selected, or can directly place action scores or Q-values, or can parameterize a distribution from which action scores or Q-values ​​are selected. The reinforcement learning training method can be deterministic or stochastic; it may be a distributed method, such as with multiple actors and / or learners.

[0024] Thus, the techniques described herein can be used in conjunction with reinforcement learning systems, including, for example: policy-based systems (e.g., Advantage Actor-Critic (A3C) systems, Mnih et al., 2016, or variants thereof), which directly parameterize the policy and, optionally, the value function; Q-learning systems, such as Deep Q-Learning Network (DQN) systems or Double-DQN systems, in which an approximate action-value function and, optionally, a state value, are output for determining an action; continuous control reinforcement learning systems, such as DDPG (Lillicrap et al., 2015) or variants thereof; or reinforcement learning systems based on distributional actor-critic or Q-values, such as IMPALA (Importance Weighted Actor-Learner), Esholt et al., arXiv:1802.01561, or Recurrent Replay Distribution DQN (R2D2), Kapturowski et al., ICLR, 2019.

[0025] In some embodiments, the action selection neural network system may include a state-value neural network to generate state values ​​or state-action values ​​(Q-values) for selecting or learning to select actions to be performed by the agent. This system may have the same parameters as the model neural network. In some embodiments, the action selection neural network system may include: a policy neural network (or policy head) that generates a policy output that defines a probability distribution over possible actions (e.g., a corresponding score for each action in a predetermined set) for selecting an action; and a value neural network (or value head) that generates an output representing a state value (e.g., an estimated long-term reward received by the agent starting from a state represented by an input observation).

[0026] For example, in some embodiments, the action selection neural network system may include or receive inputs from one or more recurrent neural network (RNN) layers. These may be shared by both the action selection neural network system and the hindsight value neural network and / or the state value neural network. For example, the internal state or hidden layer representation of one or more RNN layers may be provided as input to each of the action selection neural network system and the hindsight value neural network and / or the state value neural network. The one or more RNN layers may receive data from each observation representing the state of the environment. In an embodiment, the RNN layers are not trained using hindsight.

[0027] In some embodiments, the action selection neural network system can implement a Q-learning process in which an action score or Q-value is determined for each discrete action in a discrete action set. The action scores can be output from the action selection neural network system or can be sampled from a distribution parameterized by the action selection neural network system. The hindsight value neural network and / or the state value neural network can then be configured to generate a corresponding set of action scores. The training target for time step t can be determined from the corresponding set of action scores, for example, by determining a single scalar value representing the action score or by determining the corresponding difference between each action score (or distribution) from the action selection neural network system and the corresponding action score (or distribution) from the hindsight value neural network.

[0028] In some embodiments, the action selection neural network system can implement an actor-critic process, wherein the action selection neural network system is configured to generate an action selection policy output and a value estimate output (critic). The value estimate represents an estimate of the reward (e.g., a time-discounted reward) resulting from selecting a future action performed by the agent according to the current values ​​of the action selection network parameters, given the current state of the environment. For example, the estimated n-step-forward reward can be given by:

[0029]

[0030] where γ is a discount factor between 0 and 1, r t+i is the reward received at time step t+i, and V(o t+n ) refers to the value estimate at time step t+n. The time-discounted return can be used as a training objective for hindsight-valued neural networks and state-valued neural networks.

[0031] In the case where the action selection neural network system implements an actor-critic-based approach (or other approaches), the action selection neural network system can provide separate outputs defining state values ​​and action selections (policies). In such a system, observations representing the current state of the environment and the outputs of the model neural network are processed by the action selection neural network system to generate state value and action selection (policy) outputs. For example, observations can be processed by one or more RNN layers and then combined with the outputs of the model neural network in one or more subsequent neural network layers to generate action selection outputs, such as action selection (policy) outputs (as well as state value outputs).

[0032] Broadly speaking, the hindsight value neural network and the state value neural network can each be configured to generate a vector of state values ​​or state-action values ​​(Q-values), which depends on the output of the hindsight model neural network representing the state trajectory and the output of the model neural network representing the predicted state trajectory, respectively.

[0033] Typically, observations used in embodiments of the system may be pre-processed, for example, by convolutional or other neural network methods.

[0034] Once the model neural network is trained, the hindsight model neural network can be omitted. That is, the system can be used to perform tasks without the hindsight model neural network.

[0035] Thus, a method for selecting an action to be performed by an agent performing a task in an environment may include receiving an observation representing a current state of the environment. The method may further include processing the observation using a model neural network to generate a model neural network output, such as a multi-dimensional feature vector, representing a predicted state trajectory, wherein the predicted state trajectory includes a series of k predicted future states of the environment starting from the current state. The method may further include processing the observation representing the current state of the environment and the model neural network output to generate an action selection output for selecting an action to be performed by the agent performing the task.

[0036] Additional features of the method may be as previously described, but omitting the hindsight model neural network.

[0037] Also provided is a neural network computer system for performing a task, comprising a combination of a (trained) action selection neural network system and a (trained) model neural network.

[0038] In another aspect, a reinforcement learning neural network system is provided, comprising an action selection neural network system for selecting an action to be performed by an agent performing a task in an environment. The action selection neural network system may be configured to receive input data from i) an observation representing a current state of the environment, and / or ii) an output of a model neural network, and to process the input data according to action selection neural network system parameters to generate an action selection output for selecting an action to be performed by the agent. The system may also include the model neural network. The model neural network may be configured to receive input from an observation representing a current state of the environment. The output of the model neural network may represent a predicted state trajectory, the predicted state trajectory comprising a series of k predicted future states of the environment starting from the current state.

[0039] The system can be configured to train a hindsight model neural network by processing observations representing a state of the environment at time step t and at a series of k subsequent time steps and adjusting parameters of the hindsight model neural network using a training objective at time step t, the hindsight model neural network having an output representing a state trajectory, wherein the state trajectory includes a series of k states of the environment starting from the state of the environment at time step t. The system can also be configured to train the output of the model neural network to approximate the output of the hindsight model neural network.

[0040] In some embodiments, the environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment, such as a robot, vehicle, or other mechanical agent. The environment can also be a simulation of a real-world environment and the agent can be a simulation of a mechanical agent, in which case the system / method can be trained in the simulated environment and then deployed to control an agent in a real-world environment.

[0041] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following advantages.

[0042] Compared to previous techniques, reinforcement learning methods and systems using these techniques may be able to learn faster and more efficiently using less computational power. Model neural networks can learn from richer datasets (i.e., trajectory observations) with hindsight and may be able to selectively learn features relevant to estimating the state value of the environment. This is particularly true when the model neural network learns a reduced-dimensional representation of the observations. Using a reduced-dimensional model can also simplify the learning process.

[0043] Broadly speaking, the hindsight model neural network learns the features of future observations that are most useful for predicting the value of the current state and passes these features to the model neural network. As a result, learning is faster and requires less memory and computational power than some other techniques. The described systems and methods may be able to learn to perform tasks that other techniques cannot perform, or learn to perform tasks better than other techniques, especially in complex environments. The described systems and methods are general because they can be applied to a wide range of reinforcement learning tasks and do not require hand-crafted auxiliary rewards.

[0044] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the detailed description below. Other features, aspects, and advantages of the subject matter will become apparent from the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A reinforcement learning system implementing hindsight modeling is shown.

[0046] Figure 2 It is used for training Figure 1 Flowchart of an example process for a reinforcement learning system.

[0047] Figures 3a to 3d Schematically illustrates Figure 1 Training and operation of an example reinforcement learning system.

[0048] Figure 4 is based on Figures 3a to 3d Flowchart of an example process for training a reinforcement learning system.

[0049] Figure 5a and Figure 5b Figure 3 shows the relationship between the actor-critic reinforcement learning system and the Q-learning reinforcement learning system. Figure 1 Example neural network architecture for a reinforcement learning system.

[0050] In the drawings, like reference numerals indicate like elements. DETAILED DESCRIPTION

[0051] This specification describes a neural network-based reinforcement learning system and method. During training, the system / method models those aspects of the environment in which the system operates that, with the benefit of hindsight, are identified as being important for achieving a desired outcome, as indicated by reward signals from the environment.

[0052] In an embodiment, rather than simply learning from scalar rewards, or attempting to determine a complete model of the environment, the system / method learns to model a small number of features that characterize rewards from a trajectory of future states. During training, this future trajectory is known because it has already occurred, that is, the model uses hindsight. However, at the specific time when the system / method is controlling the agent to take an action, this future information is not available. Therefore, a second model is trained to approximate the hindsight model, but using only current (and past) information, and this second model is used to select an action.

[0053] In this specification, the model using hindsight is implemented by a hindsight model neural network, and the second model is implemented by a model neural network. At the current time step, the hindsight model neural network receives data from future observations as additional input and learns features of these observations that are useful for learning the training target (e.g., state value estimation or state-action (Q) value estimation). The model neural network learns to predict these features from one or more observations available at the current time step.

[0054] Figure 1 A reinforcement learning system 100 is shown, which can be implemented as a computer program on one or more computers in one or more locations. At each of a plurality of time steps t, the reinforcement learning system 100 selects an action a to be performed by the agent 102 in the environment 104. t At each time step, the reinforcement learning system 100 receives and processes data representing the state of the environment for selecting an action, which is referred to in this description as observations. tObservations may include images and / or other sensor data of, or input data from, the environment.

[0055] The reinforcement learning system 100 may also receive t The reward r t Generally speaking, rewards are numerical and can be based on any event or aspect of the environment. For example, the reward r t It may indicate whether the agent 106 has completed a task (eg, manipulated a task, or navigated to a target location in an environment) or the progress of the agent 106 toward completing a task.

[0056] In some embodiments, the environment is a real-world environment, or a simulation of a real-world environment. The agent may include a mechanical agent that interacts with the real-world environment, or a simulation of such a mechanical agent, or a control system for a mechanical agent.

[0057] For example, an agent can include a control system for an autonomous or semi-autonomous vehicle that navigates in an environment. In these embodiments, an action can be a possible control input for controlling the vehicle, and the result that the agent attempts to achieve is to meet the goal of the vehicle navigating in the real-world environment. For example, the goal can include one or more goals, such as: reaching the destination, ensuring the safety of any occupants of the vehicle, minimizing the energy used to reach the destination, and maximizing the comfort of the occupants. As another example, the agent can be a robot or other mechanical agent that interacts with the environment to achieve a specific task, such as locating an object of interest in the environment, or picking up an object of interest or moving an object of interest to a specified location in the environment.

[0058] In these embodiments, actions may be possible control inputs for controlling a robot (i.e., controlling the physical behavior of a robot or vehicle). For example, actions may be control inputs to control a robot, such as torques or higher-level control commands to a joint of a robot; or to control an autonomous or semi-autonomous land or air or sea vehicle, such as torques or higher-level control commands to a control surface or other control element of a vehicle; or for example motor control data. In other words, actions may include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or part of another mechanical agent. Action data may include data for these actions and / or electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within an environment, the control of which affects the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land or air or sea vehicle, actions may include actions to control navigation (e.g., steering) and movement (e.g., braking and / or acceleration of the vehicle).

[0059] In general, observations can include, for example, one or more of images, object position data, and sensor data to capture observations as the agent interacts with the environment, such as sensor data from images, distance or position sensors, or from actuators. In the case of a robot or other mechanical agent or vehicle, observations can similarly include one or more of the following: position, linear or angular velocity, force, torque, or acceleration, and the global or relative pose of one or more parts of the agent. Observations can be defined in 1, 2, or 3 dimensions and can be absolute and / or relative observations. For example, in the case of a robot, observations can include data characterizing the current state of the robot, such as one or more of the following: joint position, joint velocity, joint force, torque, or acceleration, and the global or relative pose of a part of the robot (e.g., an arm) and / or an item held by the robot. Observations can also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or image or video data, such as from a camera or lidar sensor, such as data from a sensor of the agent, or data from a sensor positioned separately from the agent in the environment.

[0060] Rewards, i.e., external rewards from the environment, can include, for example, one or more rewards for approaching or achieving one or more target positions, one or more target poses, or one or more other target configurations. For example, for a robot, rewards may depend on joint orientations (angles) or velocities, end effector positions, center of mass positions, or positions and / or orientations of groups of body parts. Costs (i.e., negative rewards) can be similarly defined, for example, depending on the forces applied when interacting with an object, energy usage, or the positions of robot body parts.

[0061] The system / method can be used to train vehicles or robots to perform tasks such as warehouse, logistics, or factory automation tasks, such as collecting, placing, or moving stored goods or collecting, placing, or moving goods or parts of goods during the manufacturing of goods; or the learning tasks can include package delivery control tasks. Actions can include actions related to steering or other directional control actions, and observations can include observations of the position or movement of other vehicles or robots.

[0062] The robot or vehicle can be trained in simulation before being used in a real-world environment. As described later, once the system is trained, the hindsight model neural network is no longer needed and can be omitted.

[0063] In some embodiments, the real-world environment may be a manufacturing plant or service facility, the observations may relate to the operation of the plant or facility, for example, to resource usage (such as power consumption), and the agent may control actions or operations in the plant / facility to, for example, reduce resource usage.

[0064] Thus, in some applications, the agent may be an electronic agent, and the observations may include data from one or more sensors monitoring a portion of a plant or service facility, such as current, voltage, power, temperature, and other sensor and / or electronic signals representing the functionality of electronic and / or mechanical items of equipment. The agent may control actions in a real-world environment that includes items of equipment, such as actions within a facility such as a data center, server farm, or grid main power or water distribution system, or control actions within a manufacturing plant or service facility, or control actions within a power generation facility such as a solar or wind farm. Observations may relate to the operation of the plant or facility, for example, they may include observations of equipment power or water usage, observations of power generation or distribution control, or observations of resource use or waste generation. Actions may include actions that control or impose operating conditions on items of equipment in the plant / facility and / or actions that result in changes to operating settings of the plant / facility, such as adjusting or turning on / off components of the plant / facility. Learning tasks may be control tasks with corresponding rewards, such as control of resource usage (e.g., water or electricity); control of environmental impacts; control of electrical or other power consumption; control of heating, cooling, or temperature; and general control of items within the facility. The reward may include one or more rewards that are contingent upon performance of such a control task.

[0065] In some other embodiments, the real-world environment may be a renewable energy plant, the observations may relate to the operation of the plant, for example to maximize current or future planned power generation, and the agent may control actions or operations in the plant to achieve this.

[0066] Thus, generally speaking, in embodiments, the agent may be a mechanical or electronic agent, and the actions may include control inputs to control the mechanical or electronic agent. Observations may be obtained from sensors such as image sensors, and / or they may be obtained from electrical or mechanical signals from the agent.

[0067] In some other embodiments, the environment is a real-world environment, and the agent is a computer system that generates outputs to present to a user. For example, the environment can be a patient diagnostic environment, such that each state is a corresponding patient state of the patient, i.e., as reflected by health status data that characterize the patient's health status, and the agent can be a computer system for suggesting treatments to the patient. In this example, the actions in the action set are possible medical treatments for the patient, and the results to be achieved can include one or more of maintaining the patient's current health status, improving the patient's current health status, minimizing the patient's medical expenses, etc. The observations can include data from one or more sensors, such as image sensors or biomarker sensors, and / or can include processed text, such as from medical records.

[0068] As another example, the environment can be a chemical synthesis or protein folding environment, such that each state is a corresponding state of a protein chain or one or more intermediates or precursor chemicals, and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, the actions are possible folding actions for folding the protein chain or actions for assembling precursor chemicals / intermediates, and the results to be achieved can include, for example, folding the protein so that the protein is stable and so that the protein achieves a specific biological function, or providing an efficient synthesis route for the chemical. As another example, the agent can be a mechanical agent that performs or controls protein folding actions or chemical synthesis steps automatically selected by the system without the need for human interaction. Observations can include direct or indirect observations of the state of the protein or chemical / intermediate / precursor, and / or can be obtained from simulations.

[0069] In some applications, the agent can be a static or mobile software agent, i.e., a computer program configured to operate autonomously and / or in conjunction with other software agents or humans to perform tasks. For example, the environment can be an integrated circuit routing environment, and the system can be configured to learn to perform routing tasks to route interconnects of an integrated circuit, such as an ASIC. The reward (or cost) may then depend on one or more routing metrics, such as interconnect resistance, capacitance, impedance, loss, speed or propagation delay, physical line parameters (e.g., width, thickness, or geometry), and design rules. Observations can be observations of component positions and interconnects; these actions can include component placement actions, such as to define component positions or orientations, and / or interconnect routing actions, such as interconnect selection and / or placement actions. Thus, routing tasks can include placing components, i.e., determining the positions and / or orientations of components of the integrated circuit, and / or determining the routing of interconnects between components. Once the routing task has been completed, the integrated circuit, such as the ASIC, can be manufactured according to the determined placement and / or routing.

[0070] In some applications, the environment can be a packet communication network environment, and the agent can include a router to route packets on the communication network. Actions can include packet routing actions, and observations can include, for example, observations of a routing table that includes routing metrics, such as routing path length, bandwidth, load, number of hops, path cost, delay, maximum transmission unit (MTU), and reliability. Learning tasks can include packet routing tasks with rewards / costs to maximize or minimize one or more routing metrics. In some other applications, the agent is a software agent that manages the allocation of computing tasks across computing resources, such as on mobile devices and / or in data centers. In these embodiments, observations can include observations of computing resources, such as computing and / or memory capacity, or Internet-accessible resources; and actions and related tasks can include assigning computing tasks to specific computing resources. Rewards may depend on, for example, the utilization of computing resources, power, bandwidth, and computing speed.

[0071] In some other applications, the environment is an Internet or mobile communication environment, and the agent is a software agent that manages personalized recommendations for users. Observations may include actions previously taken by users (characteristics of said actions); tasks may include the action of recommending items (e.g., content items) to users. Rewards may include an estimated likelihood that the user will respond positively to the recommended (content) items, or the number of recommendations received by the user (optionally within a time span); costs may depend on the suitability of one or more recommended items, the cost of the recommended (one or more) items. As another example, the action may include presenting an advertisement, the observation may include ad impressions or clicks or click-through rates, and the reward may characterize one or more users' previous selections of items or content.

[0072] In some other applications, the environment is a cybersecurity environment. For example, observations may include data characterizing the state of a computer network or distributed computing system, and actions may define one or more tasks to be performed, for example, by one or more other agents, to protect the computer system from cybersecurity attacks. Rewards may include one or more rewards that depend on a measure of the security of the system / environment, such as the number of attacks detected.

[0073] Generally speaking, in the above applications, where the environment is a simulated version of the real-world environment, once the system / method has been trained in simulation, it can then be applied to the real-world environment (and optionally omitting the hindsight model neural network). That is, the control signals generated by the system / method can be used to control the agent to perform a task in the real-world environment in response to observations from the real-world environment. Optionally, the system / method can continue to train in the real-world environment based on one or more rewards from the real-world environment.

[0074] In some other embodiments, environment is a simulated environment, and agent is implemented as one or more computer programs interacting with the simulated environment. For example, simulated environment can be a virtual environment in which a user competes with a computerized agent to achieve a goal, and agent is a computerized agent. In this example, the action in the middle of the action set is a possible action that can be performed by the computerized agent, and the result to be achieved can be, for example, winning the competition with the user. In some embodiments, environment is a simulated environment, and agent is implemented as one or more computer programs interacting with the simulated environment. For example, simulated environment can be a video game, and agent can be a simulated user playing a video game. As another example, simulated environment can be a motion simulation environment, such as driving simulation or flight simulation, and agent is a simulated vehicle navigating in the motion simulation. In these embodiments, action can be the control input of controlling the simulated user or simulated vehicle.

[0075] Optionally, in any of the above embodiments, observations at any given time step may include data from previous time steps that may be useful in characterizing the environment, such as actions performed at previous time steps, rewards received at previous time steps, and so on.

[0076] Reference again Figure 1 , the reinforcement learning system 100 includes a hindsight model neural network 130 that receives and processes data representing a state trajectory, the state trajectory comprising a sequence of k states of the environment starting from the state of the environment at the current time step t, and provides a corresponding feature vector output φ. The received and processed data is obtained from one or a series of observations occurring after time t, such as o t+1 ...o t+k , which represents the corresponding state of the environment.

[0077] The training engine 150 is configured to adjust parameters of the hindsight model neural network using the training objective of the reinforcement learning system 100. For example, the output of the hindsight model neural network can be used to determine an estimate of the training objective, and the parameters of the hindsight model neural network can be adjusted, such as by backpropagation, to improve the estimate.

[0078] The reinforcement learning system 100 also includes a model neural network 120 that receives observations representing the current state of the environment. t The training engine 150 is configured to train the feature vector output of the model neural network 120 based on, for example, a loss function (such as square loss or cross entropy loss). The output of the model neural network is approximated with hindsight, thereby measuring the difference between the two outputs above.

[0079] Thus, the feature vector output of the training model neural network 120 represents a predicted state trajectory, which includes a series of k predicted future states of the environment starting from the current state. Because the hindsight model neural network is trained using the training target, the feature vector output of the model neural network 120 also learns features that can be used to estimate this target.

[0080] In an embodiment, the training objective comprises a state value or state-action value for the current state of the environment. When following the action-selection policy of the reinforcement learning system 100, this may be the reward at time step t starting from the current state (state value) or starting from the current state (state-action value) of a specific action that is first performed. Generally speaking, the reward is an estimate of the cumulative reward (e.g., the long-term time-discounted sum of future rewards) that the agent would receive from the environment if it interacted with the environment by performing the action selected by the reinforcement learning system.

[0081] The reinforcement learning system 100 includes an action selection neural network system 110. This system receives observations from a model neural network 120. t and feature vector output Both, and use this information to generate an action selection output to select the action to be performed by the agent.

[0082] The action selection neural network system 110 may implement any reinforcement learning technique, such as Q-learning or actor-critic techniques.

[0083] For example, the action selection output may include a vector of Q values, one for each action in the set of possible actions that can be performed by the agent. The action with the largest Q value can then be selected, or the Q values ​​can be processed, for example using a soft-max function, to generate a corresponding probability value for each possible action, which can be used to select an action to be performed by the agent.

[0084] In the actor-critic approach, the action selection output can include an action selection policy output that includes a corresponding numeric probability value for each action in a set of possible actions that can be performed by the agent at the time step. The action to be performed by the agent can then be determined by selecting the action with the highest probability value, or by sampling the actions according to their probability values. As another example, the action selection policy output can directly define the action to be performed by the agent, such as by indexing an action in a set of possible actions that can be performed by the agent, or by providing a continuous-valued output that defines the value of a continuous valued action to be performed by the agent. In the actor-critic approach, the estimated state value is used as a critic when adjusting the parameters of the action selection neural network system during training.

[0085] exist Figure 1 In , only the hindsight model neural network 130 is used during training, so the hindsight model neural network and Figure 1 Other elements shown in dashed lines may be omitted from the trained system / method.

[0086] Figure 2 It is used for training Figure 1 Flowchart of an example process for a reinforcement learning system for an agent. The process receives observations and rewards (step 200) and processes data derived from the current observation using the model neural network 120 (step 202). The output of the model neural network 120 is processed by the action selection neural network system 110 in conjunction with data derived from the current observation to select an action to be performed by the agent (step 204).

[0087] Data representing the future trajectory of the state of the environment within k time steps, derived from one or more future observations, is processed by hindsight model neural network 130 (step 206). This can be accomplished, for example, by delaying the processing of the current observation by k time steps. The process then uses the training objective for the current time step to adjust the parameters of hindsight model neural network 130 (step 208) and the parameters of model neural network 120 so that the output provided by model neural network 120 approximates the output of hindsight model neural network 130 (step 210).

[0088] Repeated execution Figure 2The process may be continued, for example, until a termination criterion is reached, such as training for a number of time steps or obtaining a reward indicating task completion, or indefinitely. The steps may be performed in an order different from that shown. After training, steps 206 to 210 may be omitted.

[0089] Figures 3a to 3d Schematically illustrates Figure 1 Example training and operation of the reinforcement learning system 100. Figure 3a schematically illustrates an example architecture of a reinforcement learning system, and Figures 3b to 3d The training system is shown in the figure. Figures 3a to 3d In [1], nodes indicate neural network outputs, and edges represent learnable transformations, i.e., one or more neural network layers, that lead to those outputs. Edges are bold when learned, and the slash zero indicates that the gradient stops during backpropagation.

[0090] Figures 3a to 3d The specific example of [ 0 ] is related to actor-critic reinforcement learning systems and illustrates predicted state-values. Q-learning systems are trained and operated similarly, but predict state-action values.

[0091] Not present Figures 3a to 3d The action selection neural network system is explicitly shown in . In the actor-critic reinforcement learning system, an action selection neural network (not shown) may be included; this may share parameters with the state value neural network described below. In the Q learning system, the vector of Q values Can be used to select actions.

[0092] exist Figures 3a to 3d For a specific example, observe o t The observations may be pre-processed by one or more convolutional neural network (CNN) layers (not shown). The RNN layer(s) are based on the current observation and the previous internal state h t-1 Provide status indication h t , the state represents h t Often called the internal or hidden state representation. This can be useful when (usually) the environment is only partially observable.

[0093] exist Figures 3a to 3d In, h t and h t+k denote the internal state of the RNN layer(s) at time t and t+k, respectively; φ t denotes the output of the hindsight model neural network 130 at time t, and Indicates the output of the model neural network 120 at time t. The hindsight model neural network 130 processes the input ht+k To determine φ t , and the model neural network 120 processes the input h t To determine

[0094] During training, the system uses a hindsight value neural network to implement a hindsight value function to determine an estimated hindsight value of the state of the environment at time step t. Depend on Figures 3a to 3d The hindsight value of the edge indication in the neural network is expressed as (not labeled in the figure) and has neural network parameters θ1. Hindsight value neural network processing Hindsight model neural network feature vector output φ t and from observation o t Data (in Figures 3a to 3d The middle is h t ) to determine If the hindsight model neural network has parameters θ2, then It can be given by the following formula:

[0095]

[0096] in Any of the future observations, actions, and rewards after time t may be included, for example, k observations after time t.

[0097] The system also has a state-value neural network to determine a model-based value estimate v of the state of the environment at time step t m .Depend on Figures 3a to 3d The state value indicated by the edge in the neural network is represented as (not labeled in the figure) and has neural network parameters η1. In an embodiment, the value estimate For training action selection neural network systems (and with input Action selection neural network system). State value neural network processing model neural network feature vector output and from observation o t Data (in Figures 3a to 3d The middle is h t ) to determine If the model neural network has parameter η2, then It can be given by the following formula:

[0098]

[0099] In a system using Q-learning, and Can generate state-action value, i.e. Q value and The corresponding vector, rather than the scalar state value and Alternatively, kind An output can be generated that parameterizes the distribution of Q values ​​from which values ​​are randomly selected and

[0100] Although it can be seen from the above equation that except for h t In addition to China, There is no additional information in Still, it can provide a better training signal because it can benefit from training on the richer signal in hindsight. In addition, the use of the described model and the hindsight model can force information about future trajectories to pass through the bottleneck representation of future trajectories, so that the learned feature vector identifies those future / predicted future features that are most useful in predicting the reward of the trajectory at time t.

[0101] In an embodiment, the feature vector outputs from the hindsight model neural network and the learning of the model neural network have lower dimensions than their inputs, e.g., less than 20. This can facilitate learning. Similarly, a limited view of future observations (i.e., the number of steps k) is also useful, and the system look-ahead may be limited to, e.g., no more than 20 steps, e.g., about 5 time steps. However, the system does not appear to be particularly sensitive to the exact values ​​of these hyperparameters.

[0102] Figure 3b Highlighting the loss function for training the reinforcement learning system 100 A subset of . Value function loss Can be a value estimate Any measure of the difference (e.g., squared difference (mean squared error)) from a reinforcement learning training target. The training target can be the state value determined from the reward at time step t. For example, in an IMPALA-based reinforcement learning system (ibid), the training target can be the V-trace target.

[0103] In the Q-learning reinforcement learning system, the value function loss Can be a specific action Any measure of the difference (e.g., squared difference) between the state-action value estimate of t and a reinforcement learning training target, which may be the state-action value of a particular action determined from the reward at time step t. The particular action may be an action (or one of the action trajectories) sampled from a memory storing a sequence of experience tuples, each experience tuple corresponding to a corresponding time step. The experience tuple corresponding to time step t may include observations o t, select the action a to be performed by the agent at that time step t , the reward r received after the agent performs the chosen action t , and subsequent observations o t+1 (When storing a sequence of tuples, o can be omitted t+1 , such as in R2D2 ibid). For example, in a reinforcement learning system based on R2D2 (ibid), the training objective can be the n-step reward Q-objective defined in the paper by Kapturowski et al.

[0104] As in Figure 3b As shown in , when using the value function loss When training the reinforcement learning system 100, the training engine 150 backpropagates the gradient of the loss to update the state value neural network The neural network parameters η1 and also update one or more RNN layers (generating h t ) and the neural network parameters of one or more CNN layers (if any). The parameters η2 of the model neural network do not use the value function loss Training is performed, so only for the state value neural network, is just an additional input. The quantity denoted by the upper dash is considered non-differentiable, i.e. where the gradient stops, which can be summarized as

[0105] Figure 3c Highlighting the hindsight value function loss for training the reinforcement learning system 100 A subset of . Hindsight value function loss Can be an estimated hindsight value Any measure of difference (e.g., squared difference) from the reinforcement learning training objective. That is, the same reinforcement learning training objective used for the value function loss and hindsight value function loss Therefore, the estimated hindsight value Sum Estimation Aims to estimate the same training objective, but uses hindsight for the estimated hindsight value

[0106] As in Figure 3c As shown in , when using the hindsight value function loss When training the reinforcement learning system 100, the training engine 150 backpropagates the gradient of the loss to update the hindsight value neural network The neural network parameters θ1 and also update the hindsight model neural network The neural network parameters θ2. In an embodiment, the parameters of the (one or more) RNN layers are not lost using the hindsight value function. Conduct training.

[0107] Therefore, hindsight models neural network learning using observations, more specifically using RNN internal states such as h t+k The data represented by derives from the future observation trajectory from the current time step t, using hindsight to model features of the observations that can be used to estimate the training target, i.e., to predict the state value (or state-action value). Again using the horizontal notation, this can be summarized as

[0108]

[0109] Figure 3d Highlights the reinforcement learning system 100 for training model losses The model loss It can be the difference between the feature vector output of the hindsight model neural network 130 and the feature vector output of the model neural network 120, that is, φ t and Any measure of the difference between . For example, this could include φ t and The squared differences between the components of (omitting the t-correlations for clarity),

[0110]

[0111] As another example, we can use the cross entropy loss, in which case the feature vector output φ t and (i.e. in and ) can be viewed as logarithmic vectors. Softmax can be applied to each vector to transform them into probability distributions. and To determine the cross entropy loss in yes Cross entropy with respect to p.

[0112] like Figure 3d As shown in , when using the model loss When training the reinforcement learning system 100, the training engine 150 backpropagates the gradient of the loss to update the model neural network Optionally, the gradients can also be back-propagated to the RNN layer(s), and in the presence of the CNN layer(s), the observation encoding inputs to the model neural network are trained. Thus, the model neural network 120 learns to use the current observation to predict the feature vector output by the hindsight model neural network 130.

[0113] Combined loss for reinforcement learning system 100 It can be expressed as

[0114]

[0115] where α and β are weights, where for example α < β, for updating at a decreasing rate In order to model Hindsight model with time to adapt to changes t .

[0116] In the case where the action selection neural network system 110 has an action selection policy output as previously described, for example, in an actor-critic reinforcement learning system, the combined loss An actor loss can be included to update the parameters of the action selection neural network, which provides the action selection policy output. For example, the actor loss can depend on the product of the state value function and the gradient logπ with respect to the parameters of the action selection neural network, where π is the action selection policy output. Optionally, an entropy loss (which is negative to reduce the cost of high entropy) can also be included, for example depending on the sum of -π·logπ over the available actions or some variant thereof, to encourage exploration and reduce the risk of premature convergence during training. Figure 4 is based on Figures 3a to 3d Flowchart of an example process for training an example reinforcement learning system 100. Figure 4 The steps correspond to Figures 3b to 3d The steps shown in and can be performed in any order; optionally, one or more steps can be repeated before performing another step. For example, the steps can be repeated indefinitely or until a termination criterion is reached.

[0117] Thus, the reinforcement learning system 100 receives observations and rewards (step 400) and processes the observations using (one or more) CNN layers (if present), (one or more) RNN layers, and the action selection neural network system 110 to select an action to be performed by the agent (step 402). The reinforcement learning system 100 then determines the value function (or state-value function) loss as described above and uses it to train the state-value network and (one or more) RNN layers and CNN layers (step 404). The reinforcement learning system 100 also determines the hindsight value function (or state-value function) loss as described above and uses it to train the hindsight value neural network and the hindsight model neural network (step 406). The reinforcement learning system 100 also determines the model loss as described above and uses it to train the model neural network and, optionally, (one or more) RNN layers and (one or more) CNN layers. Training can be performed offline (e.g., using stored experience tuples) or online by delaying the calculation of the hindsight value function (or state-value function) loss and model loss by k time steps.

[0118] In general, the reinforcement learning system 100 can be implemented by any suitable neural network architecture, such as a neural network architecture including one or more of convolutional layers, fully connected layers, or recurrent layers.

[0119] Figure 5a and Figure 5b Example neural network architectures for actor-critic reinforcement learning systems 100 (e.g., IMPALA ibid) or Q-learning reinforcement learning systems 100 (e.g., R2D2 ibid), respectively, are illustrated, where the neural networks are represented by the previously used notation. In practice, the neural networks may share some parameters. The Q-learning reinforcement learning system includes a memory 502 for storing sequences of experience tuples for training.

[0120] For a system of one or more computers to be configured to perform a specific operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a specific operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0121] Embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly implemented computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by, or control of the operation of, a data processing device. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver device for execution by the data processing device. A computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. However, a computer storage medium is not a propagated signal.

[0122] The term "data processing apparatus" encompasses all kinds of equipment, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. An apparatus may include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0123] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document) in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing portions of one or more modules, subroutines, or code). A computer program may be deployed to execute on one computer or on multiple computers, located at one location or distributed across multiple locations and interconnected by a communications network.

[0124] As used in this specification, an "engine" or "software engine" refers to a software-implemented input / output system that provides outputs that are distinct from inputs. An engine can be a coded functional block, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine can be implemented on any suitable type of computing device, such as a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smartphone, or other fixed or portable device including one or more processors and a computer-readable medium. In addition, two or more engines can be implemented on the same computing device, or on different computing devices.

[0125] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and the device can be implemented as, dedicated logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)). For example, the processes and logic flows can be performed by, and the device can be implemented as, a graphics processing unit (GPU).

[0126] For example, a computer suitable for executing a computer program can be based on a general or special microprocessor or both, or based on a central processing unit of any other type. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic element of a computer is a central processing unit for executing or implementing instructions, and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from the one or more large-capacity storage devices or to transfer data to the one or more large-capacity storage devices, or both. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, for example, to name a few, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive).

[0127] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0128] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, to display information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input generated from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to a request received from the web browser.

[0129] The embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser, through which a user can interact with the embodiments of the subject matter described in this specification, or any combination of one or more of these back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication, such as any form or medium of a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0130] A computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0131] Although this specification contains many specific implementation details, these details should not be interpreted as limiting the scope of any invention or the scope of what can be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, each feature described in the context of a single embodiment may also be implemented individually in multiple embodiments or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and a claimed combination may be directed to a subcombination or a variant of a subcombination.

[0132] Similarly, although operations are depicted in a particular order in the figures, this should not be construed as requiring that such operations be performed in the particular order shown or in sequence, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0133] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As another example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain embodiments, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method for reinforcement learning, comprising: Training an action selection neural network system to select actions to be performed by an agent performing a task in an environment, wherein the environment is a real-world environment or a simulation of the real-world environment, and the agent comprises a mechanical agent that interacts with the real-world environment, or a simulation of the mechanical agent that interacts with the simulation of the real-world environment, or a control system for the mechanical agent, or The environment is a real-world manufacturing plant or service facility, and the agents include mechanical or electronic agents that control actions or operations in the manufacturing plant or service facility, or The environment is a renewable energy plant, and the agents include mechanical or electronic agents that control actions or operations in the renewable energy plant, or The environment is a patient diagnostic environment and the agent comprises a computer system for recommending treatment, or The environment is a chemical synthesis or protein folding environment, and the agent comprises a computer system for determining how to fold a protein chain or synthesize a chemical, or The environment is an integrated circuit routing environment, and the agents include static or mobile software agents that perform routing tasks for routing interconnects of the integrated circuit, or The environment is a packet communication network environment, and the agent comprises a router for routing packets on the packet communication network, or The environment is a computing environment and the agent comprises a software agent that manages the distribution of computing tasks across computing resources, or The environment is an Internet or mobile communication environment, and the agent comprises a software agent that manages personalized recommendations, wherein the action selection neural network system is configured to receive input data from i) observations representing a current state of the environment, and ii) outputs of a model neural network, and to process the input data according to action selection neural network system parameters to generate an action selection output for selecting the action to be performed by the agent; and wherein the model neural network is configured to receive input derived from the observation representing a current state of the environment, and an output of the model neural network represents a predicted state trajectory, the predicted state trajectory comprising a series of k predicted future states of the environment starting from the current state; The method further comprises: training a hindsight model neural network having an output representing a state trajectory, the state trajectory comprising a series of k states of the environment starting from a state of the environment at time step t, by processing data from one or more observations representing the state of the environment at time step t and at a series of k subsequent time steps, and adjusting parameters of the hindsight model neural network using a training objective at time step t; and The output of the model neural network is trained to approximate the output of the hindsight model neural network.

2. The method according to claim 1, wherein Training the hindsight model includes processing the output of the hindsight model neural network and the observed data representing the state of the environment at the time step t using a hindsight value neural network to generate an estimated hindsight value or state-action value of the state of the environment at the time step t, and backpropagating the gradient of an objective function that depends on the difference between the estimated hindsight value or state-action value of the state of the environment at the time step t and the training target at the time step t to update parameters of the hindsight value neural network and parameters of the hindsight model neural network.

3. The method according to claim 2, wherein: The action selection neural network system includes a state-value neural network for selecting or learning to select the action to be performed by the agent.

4. The method according to claim 3, wherein: Training the action selection neural network system includes backpropagating the gradient of an objective function that depends on the difference between a state-value or state-action value of a current state of the environment determined using the state-value neural network and an estimated reward or state-action value of the current state of the environment.

5. The method according to claim 3, wherein: The action selection neural network system has the same parameters as the model neural network.

6. The method of claim 2 , further comprising providing internal states of one or more recurrent neural network (RNN) layers that receive data from the observations representing the state of the environment as input to the action selection neural network system and the hindsight value neural network.

7. The method of claim 1 , further comprising providing internal states of one or more recurrent neural network (RNN) layers that receive data from the observations representing the state of the environment as input to the hindsight model neural network and the model neural network.

8. The method according to claim 1, wherein The training target at the time step t includes a state value or a state-action value target at the time step t.

9. The method according to claim 1, wherein: The training objective at the time step t comprises the estimated reward at the time step t.

10. The method according to claim 1, wherein Training the output of the model neural network to approximate the output of the hindsight model neural network includes backpropagating the gradient of an objective function that depends on the difference between the output of the hindsight model neural network representing the state trajectory and the output of the model neural network representing the predicted state trajectory.

11. The method according to claim 1, wherein The output of the hindsight model neural network representing the state trajectory and the output of the model neural network representing the predicted state trajectory each include features of a dimensionality reduced representation of one or more observations of the environment.

12. The method according to claim 1, wherein The output of the hindsight model neural network representing the state trajectory and the output of the model neural network representing the predicted state trajectory each have a dimension that is smaller than their inputs, smaller than 20, or smaller than 10.

13. The method according to claim 1, wherein k is less than 20.

14. The method according to any one of claims 1 to 13 further includes maintaining a memory that stores data representing trajectories generated due to the interaction of the intelligent agent with the environment, each trajectory including data at each time step in a series of time steps, the data at each time step at least identifying an observation characterizing a state of the environment and a series of subsequent observations characterizing subsequent states of the environment for training the hindsight model neural network.

15. A computer-implemented method for selecting an action to be performed by an agent performing a task in an environment, in, the environment is a real-world environment or a simulation of the real-world environment, and the agent comprises a mechanical agent interacting with the real-world environment, or a simulation of the mechanical agent interacting with the simulation of the real-world environment, or a control system for the mechanical agent, or The environment is a real-world manufacturing plant or service facility, and the agents include mechanical or electronic agents that control actions or operations in the manufacturing plant or service facility, or The environment is a renewable energy plant, and the agents include mechanical or electronic agents that control actions or operations in the renewable energy plant, or The environment is a patient diagnostic environment and the agent comprises a computer system for recommending treatment, or The environment is a chemical synthesis or protein folding environment, and the agent comprises a computer system for determining how to fold a protein chain or synthesize a chemical, or The environment is an integrated circuit routing environment, and the agents include static or mobile software agents that perform routing tasks for routing interconnects of the integrated circuit, or The environment is a packet communication network environment, and the agent comprises a router for routing packets on the packet communication network, or The environment is a computing environment and the agent comprises a software agent that manages the distribution of computing tasks across computing resources, or The environment is an Internet or mobile communication environment, and the agent comprises a software agent that manages personalized recommendations, The method comprises: receiving an observation representing a current state of the environment; processing the observations using a model neural network to generate a model neural network output representing a predicted state trajectory, the predicted state trajectory comprising a sequence of k predicted future states of the environment starting from a current state; wherein the model neural network is trained so that the model neural network outputs approximate the output of a hindsight model neural network, and the output of the hindsight model neural network represents a state trajectory, the state trajectory comprising a sequence of k states of the environment starting from a state of the environment at time step t; The hindsight model neural network is trained by the following operations: processing data from one or more observations characterizing the state of the environment at the time step t and at a series of k subsequent time steps, and adjusting parameters of the hindsight model neural network using the training objective at time step t; The observations representing the current state of the environment and the model neural network output are processed to generate an action selection output for selecting the action to be performed by the agent performing the task.

16. A reinforcement learning neural network system comprising: an action selection neural network system that selects actions to be performed by an agent performing a task in an environment, wherein the environment is a real-world environment or a simulation of the real-world environment, and the agent comprises a mechanical agent that interacts with the real-world environment, or a simulation of the mechanical agent that interacts with the simulation of the real-world environment, or a control system for the mechanical agent, or The environment is a real-world manufacturing plant or service facility, and the agents include mechanical or electronic agents that control actions or operations in the manufacturing plant or service facility, or The environment is a renewable energy plant, and the agents include mechanical or electronic agents that control actions or operations in the renewable energy plant, or The environment is a patient diagnostic environment and the agent comprises a computer system for recommending treatment, or The environment is a chemical synthesis or protein folding environment, and the agent comprises a computer system for determining how to fold a protein chain or synthesize a chemical, or The environment is an integrated circuit routing environment, and the agents include static or mobile software agents that perform routing tasks for routing interconnects of the integrated circuit, or The environment is a packet communication network environment, and the agent comprises a router for routing packets on the packet communication network, or The environment is a computing environment and the agent comprises a software agent that manages the distribution of computing tasks across computing resources, or The environment is an Internet or mobile communication environment, and the agent comprises a software agent that manages personalized recommendations; wherein the action selection neural network system is configured to receive input data from i) observations representing a current state of the environment, and ii) outputs of a model neural network, and to process the input data according to action selection neural network system parameters to generate an action selection output for selecting the action to be performed by the agent; and Model neural network, wherein the model neural network is configured to receive input from the observation representing a current state of the environment, and an output of the model neural network represents a predicted state trajectory, the predicted state trajectory comprising a series of k predicted future states of the environment starting from the current state; The system is configured to: training a hindsight model neural network having an output representing a state trajectory, the state trajectory comprising a sequence of k states of the environment starting from a state of the environment at time step t, by processing observations representing the state of the environment at time step t and at a sequence of k subsequent time steps, and adjusting parameters of the hindsight model neural network using a training objective at time step t; and The output of the model neural network is trained to approximate the output of the hindsight model neural network.

17. A neural network computer system for performing a task, comprising: an action selection neural network system that selects actions to be performed by an agent performing a task in an environment, wherein the environment is a real-world environment or a simulation of the real-world environment, and the agent comprises a mechanical agent that interacts with the real-world environment, or a simulation of the mechanical agent that interacts with the simulation of the real-world environment, or a control system for the mechanical agent, or The environment is a real-world manufacturing plant or service facility, and the agents include mechanical or electronic agents that control actions or operations in the manufacturing plant or service facility, or The environment is a renewable energy plant, and the agents include mechanical or electronic agents that control actions or operations in the renewable energy plant, or The environment is a patient diagnostic environment and the agent comprises a computer system for recommending treatment, or The environment is a chemical synthesis or protein folding environment, and the agent comprises a computer system for determining how to fold a protein chain or synthesize a chemical, or The environment is an integrated circuit routing environment, and the agents include static or mobile software agents that perform routing tasks for routing interconnects of the integrated circuit, or The environment is a packet communication network environment, and the agent comprises a router for routing packets on the packet communication network, or The environment is a computing environment and the agent comprises a software agent that manages the distribution of computing tasks across computing resources, or The environment is an Internet or mobile communication environment, and the agent includes a software agent that manages personalized recommendations, and wherein the action selection neural network system is configured to receive input data from i) observations representing a current state of the environment, and ii) outputs of a model neural network, and to process the input data according to action selection neural network system parameters to generate an action selection output for selecting the action to be performed by the agent; and a model neural network, wherein the model neural network is configured to receive input from the observation representing a current state of the environment, and an output of the model neural network represents a predicted state trajectory, the predicted state trajectory comprising a series of k predicted future states of the environment starting from the current state, wherein the model neural network is trained so that the model neural network outputs approximate the output of a hindsight model neural network, and the output of the hindsight model neural network represents a state trajectory, the state trajectory comprising a sequence of k states of the environment starting from a state of the environment at time step t; The hindsight model neural network is trained by the following operations: processing data from one or more observations characterizing the state of the environment at the time step t and at a series of k subsequent time steps, and Parameters of the hindsight model neural network are adjusted using the training objective at time step t.

18. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 15.

Images