Learning an environment representation for agent control using bootstrap potential predictions

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an environment representation neural network controlling agent of a reinforcement learning system to perform a given task. In one aspect, the method includes receiving a current observation input and a future observation input, generating a future latent representation of a future state of an environment from the future observation input, processing using an environment representation neural network to generate a current internal representation of a current state of the environment, generating a predicted future latent representation from the current internal representation, evaluating an objective function measuring a difference between the future latent representation and the predicted future latent representation, and determining an update to a current value of environment representation parameters based on a determined gradient of the objective function.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 971,209, filed February 6, 2020. The disclosure of the earlier application is considered part of the disclosure of this application and is incorporated herein by reference. Technical Field

[0003] This manual relates to reinforcement learning. Background Technology

[0004] In reinforcement learning systems, agents interact with the environment by performing actions selected by the reinforcement learning system in response to observations that represent the current state of the environment.

[0005] Some reinforcement learning systems respond to a given observation by selecting an action to be performed by an agent based on the output of a neural network.

[0006] A neural network is a machine learning model that uses one or more non-linear units to predict the output in response to a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding parameter set. Summary of the Invention

[0007] This specification generally describes a reinforcement learning system for training a neural network system to control an agent interacting with an environment. The neural network system includes an environment representation neural network configured to process, at each time step, an environment representation network input that includes a representation of the current state of the environment to generate an internal representation of the current observation. The neural network system also includes an action selection policy neural network configured to process the internal representation to generate an action selection output that can be used to select an action to be performed by the agent in response to the current observation.

[0008] To improve the training of the neural network system, during the training period, the reinforcement learning system trains the neural network system and multiple auxiliary neural networks on an auxiliary prediction training task, i.e., in addition to the reinforcement learning training of the neural network system. In some implementations, the auxiliary neural networks may include (i) a partial representation neural network, (ii) a forward prediction neural network, (iii) a backward prediction neural network, and (iv) a latent representation neural network.

[0009] Specifically, a reinforcement learning system can train an environment representation neural network to generate predictive internal representations from the network input, which includes the current observation. That is, given an observation, the internal representation predicts the internal representations generated by the environment representation neural network from one or more other observations following the given observation in the observation sequence.

[0010] According to one aspect, a method is provided for training an environment representation neural network having multiple environment representation parameters, the environment representation neural network receiving input including observations characterizing the state of an environment interacting with an agent and generating an internal representation of the state of the environment, wherein the internal representation is used to control the agent in response to the observations. The training method includes receiving (i) a current observation input including a current observation characterizing the current state of the environment and (ii) a future observation input including a future observation characterizing future states following the current state of the environment. The method further includes generating a future potential representation of the future state of the environment from the future observation input. The method further includes using the environment representation neural network to process the current observation input according to the current values ​​of the environment representation parameters to generate a current internal representation of the current state of the environment. The method further includes generating a predicted future potential representation from the current internal representation, the predicted future potential representation being a prediction of the future potential representation of the future state of the environment. The method further includes evaluating an objective function that measures the difference between the future potential representation and the predicted future potential representation; and determining an update to the current values ​​of the environment representation parameters based on the gradient of the determined objective function. The objective function may be independent of the objective function used to train the entire reinforcement learning system.

[0011] Generating future latent representations may include using a latent embedding neural network with multiple latent embedding network parameters to process future observations in order to generate future latent representations.

[0012] Current and future observation inputs can be part of a trajectory used to train the environment to represent the neural network. Trajectories can be selected from memory, and can include sequences of observations, each associated with an action performed by an agent (or another agent) in response to an observation of the environment (or another instance of the environment), and in some cases, with a reward received by the agent. Therefore, current observation input can be an observation in the trajectory at the current time step, and future observation input can be an observation in the trajectory occurring at a subsequent time step relative to the current time step being processed.

[0013] The method may further include using an environment representation neural network and processing future observation inputs based on the current values ​​of environment representation parameters to generate a future internal representation of the future state of the environment; generating a predicted future internal representation from the future latent representation, which is a prediction of the future internal representation of the future state of the environment; and determining an update to the current values ​​of the latent embedding network parameters based on the difference between the future internal representation and the predicted future internal representation.

[0014] A fixed, uniformly random action selection strategy can be used to control the agent.

[0015] The method may further include: using an action selection policy neural network with multiple policy parameters to process the current internal representation to generate a current action selection output of the action to be performed by the specified agent in the current state; determining a reinforcement learning loss based on the current action selection output; and determining an update to the current values ​​of the policy parameters based on the reinforcement loss.

[0016] The method may also include determining the update of the current values ​​of the environment representation parameters by backpropagating the gradient of the reinforcement learning loss into the environment representation neural network.

[0017] The future state can be k steps after the current state, where k is a predetermined positive integer.

[0018] Generating a predicted future latent representation may include processing a feedforward predictive neural network input derived from the current internal representation to generate a predicted future latent representation that is a prediction of the future latent representation of the future state of the environment. The method may also include determining updates to the current values ​​of the feedforward predictive network parameters based on the gradient of the objective function relative to the feedforward predictive network parameters.

[0019] Generating a predicted future internal representation may include processing a future latent representation using a backward prediction neural network with multiple backward prediction network parameters to generate a predicted future internal representation. The method may also include determining updates to the current values ​​of the backward prediction network parameters.

[0020] The current observation input may also include prior actions performed by the agent in the prior state of the environment.

[0021] An environment representation neural network may include a partial representation neural network configured to process (i) the current internal representation and (ii) the current action performed by the agent to generate a partial future internal representation of the future state of the environment.

[0022] The forward prediction network input derived from the current internal representation can be a partial future internal representation generated by a partial representation neural network.

[0023] The method may also include backpropagating the computed gradients into the environment representation neural network to determine updates to the current values ​​of the environment representation parameters.

[0024] According to another aspect, a system is provided that includes one or more computers and one or more storage devices and storage instructions, which, when executed by the one or more computers, are operable to cause the one or more computers to perform the operations of the corresponding methods described above.

[0025] According to another aspect, a computer storage medium coded with instructions is provided, which, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding methods described above.

[0026] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages.

[0027] The reinforcement learning system described in this specification can train an environment representation neural network to learn and generate a high-quality internal representation of the state of the environment. In various cases, this internal representation can incorporate information from multiple observation modalities, including, for example, rewards, pixel images, verbal commands, and other sensory inputs such as touch, smell, sound, and temperature.

[0028] The described reinforcement learning system encourages an environment representation neural network to learn to generate more informative or predictive internal representations by performing reinforcement learning training in conjunction with one or more other auxiliary neural networks configured to perform auxiliary tasks, such as generating latent representations of the environment, forward predictions of future states, and backward predictions of past states.

[0029] In contrast, training environment representation neural networks using conventional methods is less efficient when training does not involve auxiliary tasks and focuses on generating corresponding internal representations for the current (i.e., immediate) state of the environment, or when training is based on conventional auxiliary tasks (including, for example, pixel or feature control tasks for specific observations). Specifically, this is because the reinforcement learning training systems described in this specification employ auxiliary tasks that typically require internal representations to predict future states in the latent space, while in turn requiring those latent representations to predict representations of future states.

[0030] By utilizing a trained environment representation neural network to generate a useful representation of the corresponding state of the environment, the reinforcement learning system described herein can enhance the signals to be provided to the action selection policy neural network during or after training (i.e., at runtime) to improve the quality of the action selection output. Therefore, the reinforcement learning system described herein can, for example, achieve superior performance to conventional reinforcement learning systems when controlling an agent to perform a task, by receiving more accumulated external rewards.

[0031] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0032] Figure 1 An example reinforcement learning system is shown.

[0033] Figure 2 This is a flowchart of an example process for training on a forward prediction task.

[0034] Figure 3 This is an example illustration of training on a forward prediction task.

[0035] Figure 4 These are example illustrations of generating internal representations using an environment representation neural network and generating partial internal representations using a partial representation neural network.

[0036] Figure 5 This is a flowchart of an example process for training on a backward prediction task.

[0037] Figure 6 This is an example illustration of training on a backward prediction task.

[0038] The same reference numerals and names in different figures indicate the same elements. Detailed Implementation

[0039] This specification describes a reinforcement learning system that selects actions to be performed by an agent interacting with the environment by processing data (i.e., "observations") characterizing the current state of the environment at each of multiple time steps in order to control the agent.

[0040] At each time step, the state of the environment at that time step depends on the state of the environment at the previous time step and the actions performed by the agent at the previous time step.

[0041] In some implementations, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle navigating the environment.

[0042] In these implementations, observations may include one or more of the following: images, object location data, and sensor data, to capture observations as the agent interacts with its environment, such as sensor data from images, distance or position sensors, or actuators.

[0043] For example, in the case of a robot, observations may include data characterizing the robot’s current state, such as one or more of the following: joint positions, joint velocities, joint forces, torques or accelerations (e.g., gravity-compensated torque feedback), and the global or relative orientation of the object held by the robot.

[0044] In the case of robots or other mechanical agents or vehicles, observation can similarly include one or more of the following: position, linear or angular velocity, force, torque or acceleration, and global or relative attitude of one or more parts of the agent. Observation can be defined in 1D, 2D, or 3D and can be absolute and / or relative.

[0045] Observations may also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or, for example, image or video data from a camera or LIDAR sensor, such as data from the agent's sensors or data from sensors placed separately from the agent in the environment.

[0046] In the case of electronic agents, observation may include data from one or more sensors (such as current, voltage, power, temperature and other sensors) monitoring a part of the plant or service facility, and / or electronic signals representing the function of electronic and / or mechanical equipment.

[0047] In these implementations, the action can be a control input for controlling a robot (e.g., torque of robot joints or higher-level control commands), or a control input for controlling an autonomous or semi-autonomous land, air, or sea vehicle (e.g., torque of the vehicle's control surfaces or other control elements or higher-level control commands).

[0048] In other words, actions can include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or a part of another mechanical agent. Action data may additionally or alternatively include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within an environment, wherein the control of these electronic devices affects the observed state of the environment. For example, in the case of autonomous or semi-autonomous land, air, or sea vehicles, actions can include movements for controlling the vehicle's navigation (e.g., steering) and movement (e.g., braking and / or acceleration).

[0049] In some implementations, the environment is a simulated environment, and the agent is implemented as one or more computers that interact with the simulated environment. Training the agent in a simulated environment allows the agent to learn from a large amount of simulated training data while avoiding the risks associated with training an agent in a real-world environment, such as damage to the agent due to performing inappropriately chosen actions. An agent trained in a simulated environment can then be deployed in a real-world environment.

[0050] The simulation environment can be a motion simulation environment, such as a driving simulator or a flight simulator, and the agent can be a simulated vehicle navigating in the motion simulation. In these embodiments, actions can be control inputs used to control the simulated user or the simulated vehicle.

[0051] In another example, the simulated environment could be a video game, and the agent could be a simulated user playing the video game. Typically, in the case of a simulated environment, observations can include simulated versions of one or more of the previously described observations or observation types, and actions can include simulated versions of one or more of the previously described actions or action types.

[0052] In another example, the environment can be a chemical synthesis or protein folding environment, such that each state is a corresponding state of a protein chain or one or more intermediate or precursor chemicals, and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, actions are possible folding actions for folding the protein chain or actions for assembling the precursor chemical / intermediate, and the desired outcome can include, for example, folding the protein to make it stable and enabling it to perform a specific biological function, or providing an efficient synthetic route for the chemical. As another example, the agent can be a mechanical agent that performs or controls protein folding actions selected automatically by the system (without human interaction). Observations can include direct or indirect observation of the protein's state, and / or can be derived from simulations.

[0053] In a similar manner, the environment could be a drug design environment, such that each state is a corresponding state of a potential pharmaceutical chemical, and the agent is a computer system used to determine the elements and / or synthetic pathways of the pharmaceutical chemical. For example, in a simulation, the drug / synthesis could be designed based on rewards derived from the drug's objectives. As another example, the agent could be a mechanical agent that performs or controls the synthesis of the drug.

[0054] In some applications, the agent can be a static or mobile software agent, i.e., a computer program configured to operate autonomously and / or alongside other software agents or humans to perform tasks. For example, the environment can be an integrated circuit routing environment, and the system can be configured to learn to perform routing tasks for routing interconnects of integrated circuits such as ASICs. The reward (or cost) can then depend on one or more routing metrics, such as interconnect resistance, capacitance, impedance, loss, speed or propagation delay, physical line parameters (such as width, thickness, or geometry), and design rules. Observations can be observations of component locations and interconnects; actions can include component placement actions (e.g., defining component location or orientation) and / or interconnect routing actions (e.g., interconnect selection and / or placement actions). Thus, a routing task can include placing components, i.e., determining the location and / or orientation of components in an integrated circuit, and / or determining the routing of interconnects between components. Once the routing task is completed, an integrated circuit, such as an ASIC, can be manufactured based on the determined placement and / or routing. Alternatively, the environment can be a data packet communication network environment, and the agent can be a router for routing data packets on the communication network based on network-based observations.

[0055] Typically, in a simulated environment, observations may include simulated versions of one or more of the previously described observations or observation types, and actions may include simulated versions of one or more of the previously described actions or action types.

[0056] In some other applications, agents can control actions in real-world environments, including, for example, equipment and items in data centers, power / water distribution systems, manufacturing plants, or service facilities. Observations can then be associated with the operation of the plant or facility. For example, observations could include observations of equipment's electricity or water consumption, or observations of power generation or distribution control, or observations of resource use or waste generation. Actions can include actions that control or impose operating conditions on the equipment and items of the plant / facility, and / or actions that cause changes in settings during the operation of the plant / facility, such as adjusting or turning on / off components of the plant / facility.

[0057] In some other applications, the environment is a real-world environment, and the agent manages task allocation, such as on mobile devices and / or computing resources in a data center. In these implementations, actions may include assigning tasks to specific computing resources.

[0058] Typically, in the applications described above, the environment is a simulated version of a real-world environment. Once the system / method has been trained in the simulation, it can then be applied to the real-world environment. That is, control signals generated by the system / method can be used to control the agent to perform tasks in the real-world environment in response to observations from the real-world environment. Optionally, the system / method can continue to be trained in the real-world environment based on one or more rewards from the real-world environment.

[0059] In some of the above implementations, at each time step, the system receives a reward based on the current state of the environment and the agent's actions at that time step. For example, the system may receive a reward for a given time step based on the agent's progress toward one or more objectives. For instance, the agent's objective may be navigating to a target location in the environment, and the reward may be specified by a scalar value characterizing the agent's progress toward the target location.

[0060] Figure 1 An example reinforcement learning system 100 is shown. The reinforcement learning system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, wherein the systems, components and techniques described below are implemented.

[0061] The reinforcement learning system 100 controls the agent 102 that interacts with the environment 104 by selecting an action 106 to be performed by the agent 102 and then causing the agent 102 to perform the selected action 106.

[0062] The execution of the selected action 106 by agent 102 typically causes environment 104 to transition to a new state. By repeatedly causing agent 102 to perform actions in environment 104, system 100 can control agent 102 to complete specified tasks.

[0063] The reinforcement learning system 100 includes an agent control subsystem 160, which in turn includes an environment representation neural network 110, an action selection policy neural network 130, and in some embodiments, a value prediction neural network. The reinforcement learning system 100 also includes a training engine 116 and one or more memories storing a set of network parameters 118 of the neural networks included in the agent control subsystem 160, and auxiliary neural networks 150A-D maintained by the training engine 116 during training, as will be further described below, to improve the training of the neural networks included in the agent control subsystem 160.

[0064] At each of the multiple time steps, the environment representation neural network 110 is configured to process the environment representation network input according to network parameters 118 to generate an internal representation of the current observation, the environment representation network input including (i) the current observation characterizing the current state of the environment and (ii) a previous action performed by the agent in response to a previous observation. However, the current observation may be optionally included in the previous action. The action selection policy neural network 130 is then configured to process the internal representation according to the parameters 118 (“policy parameters”) of the action selection policy neural network 130 to generate an action selection output 122.

[0065] System 100 uses action selection output 122 to control the agent, that is, to select an action 106 to be performed by the agent at the current time step according to the action selection strategy, and then make the agent execute action 106, for example, by sending a control signal directly to the agent or by sending data identifying action 106 to the agent's control system.

[0066] The following describes several examples of using action selection output 122 to select action 106 to be performed by the agent.

[0067] In one example, action selection output 122 defines a probability distribution over possible actions to be performed by the agent. For example, the action selection output may include the corresponding action probability for each of a set of possible actions that the agent can perform to interact with the environment. In another example, action selection output 122 may include distribution parameters over a set of possible actions. System 100 can use any of various action selection strategies based on action selection output 122 to select actions to be performed by the agent, for example, by sampling actions according to their probability values, or by selecting the action with the highest probability value.

[0068] In another example, the action selection output 122 responds to observation to identify the optimal action from a set of possible actions to be performed by the agent. For example, in the case of controlling a mechanical agent, the action selection output could identify the torque to be applied to one or more joints of the mechanical agent. System 100 can use any of a variety of action selection strategies based on the action selection output 122 to select the action to be performed by the agent, for example, by selecting the identified optimal action, or by adding noise to the optimal action to encourage exploration and selection of the noise-added action.

[0069] In another example, the action selection output 122 may include the corresponding Q value for each of a set of possible actions that can be performed by the agent.

[0070] The Q-value of an action is an estimate of the "return," which is generated by the agent performing an action in response to the current observation 120 and subsequently selecting a future action to be performed by agent 102 based on the current values ​​of the control neural network parameters.

[0071] Rewards refer to the cumulative measure of “rewards” 124 received by the agent, such as the total time discount of rewards.

[0072] The agent can receive a reward of 124 at each time step, where the reward 124 is specified by a scalar value and represents, for example, the agent's progress in completing a specified task.

[0073] In this example, system 100 can use any of the various action selection strategies based on action selection output 122 to select the action to be performed by the agent, for example, by selecting the action with the highest Q-value, or by mapping Q-values ​​to probabilities and sampling actions according to those probabilities. In some cases, system 100 can select the action to be performed by the agent based on an exploration strategy. For example, the exploration strategy could be... -greedy - Greedy) exploration strategy, where system 100 is based on a probability of 1- The action selection output 122 is used to select the action to be performed by the agent, and the action is randomly selected with probability. The action. In this example, It is a scalar value between 0 and 1.

[0074] In some implementations, in addition to any of the foregoing, the agent control subsystem 160 also includes a value prediction neural network configured to process, at each of a plurality of time steps, the internal representation generated by the environment representation neural network 110 and to generate a value prediction representing the value of the environment in its current state successfully performing a specified task. In other words, the value prediction is an estimate of the reward for a specified task resulting from the environment being in its current state as represented by observations; for example, if the agent control subsystem 160 is used to control the agent, it is an estimate of the sum of time-discounted rewards 124 to be received from the current state over the remainder of the task episode or over a fixed number of future time steps. This sum is called a time discount because future rewards are discounted by a fixed time discount factor γ when calculating the reward.

[0075] In some implementations, the environment representation neural network 110 is a recurrent neural network configured to receive environment representation network inputs at each of multiple time steps, thereby updating the current hidden state of the environment representation neural network 110 by processing the received environment representation network inputs. Specifically, the current hidden state of the network 110, which has already been generated by processing previous inputs, is modified by processing the currently received inputs. The environment representation network inputs include (i) a current observation 120 representing the current state of the environment and (ii) a prior action 106 performed by an agent in response to the prior observation. The updated hidden state of the environment representation neural network 110 corresponds to an internal representation of the current state of the environment 104. For example, the architecture of the environment representation neural network 110 may include a sequence of one or more layers (e.g., convolutional layers, fully connected layers, or embedding layers), followed by one or more recurrent layers (e.g., long short-term memory (LSTM) layers), and in some cases, an output layer that applies a transformation to the hidden states generated by the recurrent layers to generate the environment representation network output.

[0076] As used in this specification, hidden states, internal representations, or latent representations generally refer to intermediate data generated by a neural network from input observations, data derived from input observations, or both, and can each be represented as an ordered set of numerical values, such as vectors or matrices of numerical values. Each such intermediate data may have a lower dimension than the observation itself.

[0077] In these embodiments, the action selection policy neural network 130 and the value prediction neural network (when included) are arranged on top of the environment representation neural network 110, such that they can each receive the internal representation generated by the environment representation neural network 110 and generate the action selection output and value prediction output, respectively. For example, the hidden states generated by the recursive layers (e.g., long short-term memory (LSTM) layers) of the environment representation neural network 110 can be directly provided to the corresponding input layers of the action selection policy neural network 130 and the value prediction neural network (e.g., as intermediate outputs of the environment representation neural network 110). As another example, the environment representation network output generated from the hidden states by the output layer of the environment representation neural network 110 can be provided to the corresponding input layers of the action selection policy neural network 130 and the value prediction neural network.

[0078] The action selection strategy neural network 130 and the value prediction neural network can each be implemented as a corresponding neural network with any suitable neural network architecture (e.g., a multilayer perceptron neural network architecture) that enables it to perform the functions described therein.

[0079] The training engine 116 in the reinforcement learning system 100 trains the network included in the agent control subsystem 160 by iteratively adjusting the values ​​of network parameters 118 using reinforcement learning techniques to generate action selection outputs that maximize the expected long-term time-discounted reward received by the system 100.

[0080] For one or more time steps, system 100 stores in trajectory buffer 114 the interactions between agent 102 (or another agent) and environment 104 (or another instance of the environment) for that time step. Trajectory buffer 114 thus maintains data specifying multiple observation sequences, each associated with an action performed by the agent in response to an environmental observation, and in some cases, with a reward received by the agent. In some embodiments, a fixed, uniformly random action selection strategy can be used to control the agent interacting with the environment during trajectory data collection. In other embodiments, the agent can be controlled using agent control subsystem 160 based on the current values ​​of the parameter set 118 of the neural network included in agent control subsystem 160.

[0081] In some implementations, training engine 116 uses policy-based reinforcement learning techniques and trains environment representation neural network 110, action selection policy neural network 130, or both based on the trajectory generated. An example method for policy-based reinforcement learning is described in Volodymyr Mnih et al., “Asynchronous methods for deep reinforcement learning,” Proceedings of the 33rd International Conference on Machine Learning, 2016. Another example method for policy-based reinforcement learning is the policy gradient method. In some other implementations, training engine 116 uses off-policy reinforcement learning techniques (e.g., one-step or n-step Q-learning) based on trajectory data generated by the trajectory generation engine or when the sampling engine samples trajectory data from trajectory buffer 114, to train environment representation neural network 110, action selection policy neural network 130, or both.

[0082] In addition to training the environment representation neural network 110 and action selection policy neural network 130 included in the agent control subsystem 160 to maximize the expected long-term time-discounted reward, the system 100 can also train neural networks 110 or 130 or both by additionally training multiple auxiliary training neural networks 150A-D on the auxiliary prediction training task (i.e., in addition to reinforcement learning training of the neural networks).

[0083] Specifically, the auxiliary training neural network may include (i) a partial representation neural network, (ii) a forward prediction neural network, (iii) a backward prediction neural network, and (iv) a latent representation neural network. Furthermore, the auxiliary prediction training task may include: (i) a forward action condition prediction task for generating a predicted future latent representation of the future state of the environment from current observations of the current state of the environment, and (ii) a backward prediction task for generating a predicted future internal representation from future observations of the future state of the environment.

[0084] The training engine 116 can do this by optimizing a set of auxiliary task objective functions to iteratively update the current values ​​of the parameter set 118 of the auxiliary neural network 150A-D, and further update the current values ​​of the parameter set 118 of the environment representation neural network 110 and the action selection policy neural network 130.

[0085] Each auxiliary neural network can have an appropriate neural network structure that enables it to perform the function it describes. Each auxiliary control neural network has a corresponding set of parameters. For example, a partial representation neural network can be configured as a recurrent neural network, such as an LSTM neural network. As another example, the forward prediction neural network and the backward prediction neural network can each be configured as a corresponding feedforward neural network, and the latent embedding neural network can be configured as a convolutional neural network, i.e., a neural network with one or more convolutional network layers.

[0086] By training the environment representation neural network 110 and optionally the action selection policy neural network 130 and auxiliary neural networks 150A-D to generate predicted future latent representations from current observations (in forward prediction training) and predicted future internal representations from future observations (in backward prediction training), the reinforcement learning system 100 can determine training values ​​for the parameter set 118 that ensure the environment representation neural network 110 generates meaningful and predictive internal representations, i.e., given an observation's internal representation, it predicts the internal representations generated by the environment representation neural network from one or more other observations following the given observation in the observation sequence. These predictive internal representations, in turn, help the action selection policy neural network 130 generate more effective policy outputs, and thus more closely align with the long-term objective of the agent 102 to maximize the expected long-term time-discounted reward.

[0087] The following section will describe in more detail how reinforcement learning techniques, in addition to self-supervised representation learning, are used to train neural networks for assisted prediction training tasks.

[0088] Figure 2This is a flowchart of an example process 200 for training on a forward prediction task. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a properly programmed reinforcement learning system (e.g., Figure 1 The reinforcement learning system 100 can execute process 200.

[0089] Typically, the system can execute process 200 using a trajectory selected from the replay memory. For example, the trajectory could be one of a batch of trajectories sampled from the replay memory. The trajectory could include a sequence of observations, each observation associated with an action performed by an agent (or another agent) in response to an observation of the environment (or another instance of the environment), and in some cases, with a reward received by the agent. Specifically, the system can repeatedly execute process 200 for different future observations that have different time steps preceding a given observation (hereinafter referred to as the "current" observation) in the selected trajectory. For example, the system could execute one iteration of process 200 for each observation following the current observation in the observation sequence within the selected trajectory.

[0090] The system receives current observation input and future observation input (202).

[0091] Current observation input may include (i) a current observation representing the current state of the environment and (ii) a prior action performed by the agent in a prior state immediately preceding the current state of the environment. Future observation input may include (i) a future observation representing a future state one or more steps after the current state of the environment and (ii) a prior action performed by the agent in a prior state immediately preceding the future state of the environment. In some cases, observation may include visual data, such as images or video frames. In other cases, observation may be a multimodal observation that additionally incorporates information about text (e.g., natural language instructions, rewards, or other sensory inputs, including touch, smell, sound, or temperature).

[0092] Figure 3 This is an example illustration of training on a forward prediction task. At a given time step... t At this point, the system obtains the current observation. and characterization after the current state of the environment. k Future observation of the future state of each step ,in k It is a predetermined positive integer, and each "step" corresponds to an environment state transition as a result of an action performed by an agent in the environment.

[0093] The system generates a future latent representation of the future state of the environment from future observations of the input (204). To this end, the system can use a latent embedding neural network.f And process future observations based on the current values ​​of the parameters of the latent embedding neural network (“latent embedding parameters”). To generate potential representations for the future .

[0094] The system uses an environment to represent neural networks. It processes the current observation input based on the current values ​​of the parameters of the environment representation neural network (“environment representation parameters”) to generate a current internal representation of the current state of the environment. The current observation input includes (i) the current observation. and (ii) the preceding actions performed by the agent in the preceding environment state. .

[0095] The system then represents from the current internal context. Generate a partial future internal representation corresponding to the future state of the environment. The system can use the current internal representation. The initialization part represents the neural network. This can be achieved, for example, by enabling the partial representation neural network to have the same hidden states as the environment representation neural network, and then using the partial representation neural network thereafter. Iteratively process the data for the corresponding actions to be performed by the agent in each environment state that is specified after the current state of the environment and before the future state of the environment, as further described below.

[0096] The partial representation neural network can be configured as a recurrent neural network to receive, at each time step, a partial representation network input including the current action performed by the agent based on the current values ​​of the parameters of the partial representation neural network (“partial representation parameters”), and to update the hidden state for that time step, i.e., to modify the current hidden state of the partial representation neural network by processing the currently received partial representation network input. Because the partial representation neural network can be used to simulate state transitions of the environment without iteratively processing additional information about the environment (e.g., observations characterizing the state of the environment), the hidden state of the partial representation neural network can be referred to as a partially internal representation of the state of the environment, i.e., the opposite of the (fully) internal representation generated by the environment representation neural network 110 from both observation and action data.

[0097] Figure 4 These are example illustrations of generating internal representations using an environment representation neural network and generating partial internal representations using a partial representation neural network. Figure 2 The dashed lines in the text connect the time-aligned internal representation and the partial internal representation.

[0098] At each time step, for example, at time stept+1 At (hereinafter referred to as the "current" time step), system 100 can use an environment representation neural network ( ) to process including (i) the current observation and (ii) preceding action The environment represents the network input, and the hidden state is updated for that time step. Updated hidden state The internal representation corresponding to the current state of the environment, and which can be mathematically computed as .

[0099] Updated hidden state (Or the output generated by the output layer of the environment representation neural network from the updated hidden state) can be provided as input to the action selection policy neural network to generate action selection output, or it can be provided to the partial environment representation neural network, i.e., for initializing the hidden state of the partial environment representation neural network, so that the partial environment representation neural network can subsequently select from the updated hidden state. A partial internal representation of the future state of the generated environment. For example, the future state could be a state following the current state. k The state of each step, where k It is a predetermined positive integer, and each "step" corresponds to an environment state transition as a result of an action performed by an agent in the environment.

[0100] exist Figure 4 In the example, at the time step t+1 At this point, the system can use a portion of the environment to represent the neural network ( To process previous actions Furthermore, it determines the hidden states of the neural network that represent the partial environment at that time step. Updated hidden state The part of the current internal representation corresponding to the current state of the environment.

[0101] Based on a portion of the current internal representation, the system can iteratively use a portion of the environment representation of the neural network. This is used to generate multiple partial future internal representations of the future state of the environment. In each iteration, the partial environment representation neural network (...) It receives data as input specifying the preceding actions to be performed by the agent immediately in the preceding environment state. For example, in time steps... t+2 (when k At point (= 1), the system can use a partial environment representation of the neural network ( To handle the time steps by the agent t+1 Previous actions performed Furthermore, it determines the hidden states of the neural network that represent the partial environment at that time step. Updated hidden state This corresponds to a partial future internal representation of the future state one step after the current state of the environment.

[0102] In mathematical terms, the hidden states of a partial representation neural network, corresponding to the partial current internal representation and the partial future internal representation, can be computed separately as follows:

[0103] The system represents a partial future internal representation. The system generates a predicted future potential representation, which is a prediction of the future potential representation of the future state of the environment (208). To generate this predicted future potential representation, the system can use a portion of the future internal representation. It is provided as input to the feedforward prediction neural network. g The feedforward predictive neural network is configured to process a portion of the future internal representation based on the current values ​​of the parameters of the feedforward predictive neural network (“feedforward predictive parameters”). To generate predicted future potential representations .

[0104] The system evaluates a first auxiliary task objective function (210) that measures the difference between the potential future representation and the predicted potential future representation. The first auxiliary task objective function can train an environment representation neural network to generate an internal representation of the current state of the environment from a current observation representing the current state of the environment, which can be effectively used to predict potential representations of one or more future states of the environment after the current state of the environment.

[0105] For example, the objective function of the first auxiliary task can be evaluated as .

[0106] The system determines an update to the current values ​​of the environment representation parameters (212). To determine the update, the system can compute the gradient of the first auxiliary task objective function with respect to the forward prediction parameters and backpropagate the gradient to the partial representation parameters, and then to the environment representation parameters. The gradient operator (e.g., ...) is stopped. Figure 3 (As shown by the cross arrows in the middle) can be implemented by the system to prevent the parameter values ​​of certain components of the system (e.g., the values ​​of latent embedding parameters) from being updated during training on the forward prediction task.

[0107] Figure 5This is a flowchart of an example procedure 500 for training on a backward prediction task. For convenience, procedure 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a properly programmed reinforcement learning system (e.g., Figure 1 The reinforcement learning system 100 can execute process 500.

[0108] Similarly, as described above, the system can iteratively execute process 500 using a trajectory selected from the replay memory (e.g., the trajectory selected in process 200).

[0109] Figure 6 This is an example illustration of training on a backward prediction task.

[0110] The system uses an environment to represent neural networks. To generate a future internal representation of the future state of the environment (502). For this purpose, as... Figure 6 As shown, the system can use an environment representation neural network at each of multiple time steps. The observed input is processed based on the current values ​​of the environment representation parameters to update its hidden state; for example, the current hidden state is modified. To generate updated hidden state The observation input includes (i) the current observation representing the current state of the environment. or after the current observation k Future Observations of Each Step And until future observations characterizing the future state of the environment, and (ii) by the agent responding to the current observation. The preceding action performed Or, the agent responds to future observations. The preceding action is observed and performed immediately. The environment represents the hidden state of the final update of the neural network. The future internal representation corresponding to the future state of the environment.

[0111] The system generates a predicted future internal representation from the future potential representation, which is a prediction of the future internal representation of the future state of the environment (504). As described above, the future potential representation It can be generated by a latent embedded neural network f Observing from the future generate.

[0112] In order to generate a predicted future internal representation, the system can use the future potential representation. It is provided as input to the backward prediction neural network. g’The backward prediction neural network is configured to process future latent representations based on the current values ​​of the parameters of the backward prediction neural network (“backward prediction parameters”). To generate a predicted internal representation of the future. .

[0113] The system evaluates a second auxiliary task objective function (506) that measures the difference between the future internal representation and the predicted future internal representation. The second auxiliary task objective can train a latent embedding neural network and a backward prediction neural network to generate a predicted future internal representation from future observations of future states of the environment following the current state of the environment. This predicted future internal representation tends to match the future internal representation generated from corresponding future observation inputs using an environment representation neural network. The future observation inputs include (i) future observations representing the future state of the environment and (ii) preceding actions performed by an agent in the preceding state immediately following the future state of the environment. For example, the second auxiliary task objective function can be evaluated as follows: .

[0114] Similarly, as described above, the system determines updates to the current values ​​of the latent embedding parameters and backward prediction parameters based on the gradient of the objective function of the second auxiliary task with respect to the network parameters, for example, through backpropagation. A stopping gradient operator can be implemented by the system to prevent the values ​​of the environment representation parameters from being updated during training on the backward prediction task.

[0115] In some implementations, the system interleaves forward prediction training and backward prediction training with the system's reinforcement learning training. In short, the system's reinforcement learning training trains the action selection policy neural network to iteratively adjust the values ​​of the policy parameters using reinforcement learning techniques to generate action selection outputs that maximize the expected long-term time-discounted reward received by the system.

[0116] The system uses an action selection policy neural network to process a given internal representation based on the current value of the policy parameters, in order to generate an action selection output specifying the action to be performed by the agent in a given environment state. The given internal representation can be a hidden state generated by an environment representation neural network based on processing a given observation input, which includes (i) the current observation representing the given environment state and (ii) the preceding action to be performed by the agent in response to the immediate preceding observation of the given observation.

[0117] The system selects the output based on the current action and determines the reinforcement learning loss according to appropriate reinforcement learning training techniques, as referenced above. Figure 1 The

[0118] The system determines the update of the current value of the policy parameters by calculating the gradient of the reinforcement learning loss relative to the policy parameters. In some implementations, the system additionally determines the update of the current value of the environment representation parameters by backpropagating the gradient of the reinforcement learning loss to the environment representation parameters.

[0119] The system then continues to apply updates determined from reinforcement learning training, as well as forward and backward prediction training, to the parameter values ​​of the neural network. This involves updating some or all of the following using an appropriate gradient descent optimization method (e.g., stochastic gradient descent, RMSprop, or Adam): environment representation parameters, partial representation parameters, policy parameters, forward and backward prediction parameters, and latent embedding parameters. Alternatively, once process 200 or 500 has been performed on the entire selected trajectories in the minibatch, the system simply continues to update the current values ​​of the network parameters. Minibatches typically consist of a fixed number of trajectories, for example, 8, 16, or 32. In other words, the system continues to update the current network parameter values ​​based on the combined gradients, for example, by calculating a weighted or unweighted average of the corresponding gradients determined during a fixed number of iterations of process 200 or 500.

[0120] Typically, the system can repeatedly execute processes 200 and 500 until a termination criterion is met, such as after processes 200 or 500 have been executed a predetermined number of times, or after the gradient has converged to a specified value.

[0121] This specification uses the term "configuration" in conjunction with system and computer program components. For one or more computer systems to be configured to perform specific operations or actions, it means that software, firmware, hardware, or combinations thereof are installed on the system that, in operation, cause the system to perform the operations or actions. For one or more computer programs to be configured to perform specific operations or actions, it means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0122] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by or control of the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0123] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0124] A computer program (which may also be referred to or described as a program, software, software application, application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored as part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), stored in a single file dedicated to the program in question, or stored in multiple collaborating files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer, or on multiple computers located in one location or distributed across multiple locations and interconnected via a data communication network.

[0125] In this specification, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or not at all, and it can be stored on storage devices in one or more locations. Therefore, for example, an indexed database may include multiple collections of data, each of which can be organized and accessed differently.

[0126] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0127] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGA or ASIC), or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0128] A computer suitable for executing computer programs can be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented or incorporated therein by special-purpose logic circuitry. Typically, a computer will also include or be operatively coupled to one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, for receiving data from them, for transferring data to them, or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0129] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0130] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and, in turn, receiving response messages from the user.

[0131] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the common and computationally intensive parts of machine learning training or production, namely inference and workloads.

[0132] Machine learning frameworks (such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet) can be used to implement and deploy machine learning models.

[0133] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or application through which a user can interact with embodiments of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0134] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by means of computer programs running on their respective computers and having client-server relationships with each other. In some embodiments, for example, a server sends data (e.g., an HTML page) to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated at the user device (e.g., the result of user interaction) may be received at the server from the device.

[0135] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features characteristic of particular embodiments of a particular invention. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0136] Similarly, although operations are depicted in a specific order in the accompanying drawings and recited in a specific order in the claims, this should not be construed as requiring these operations to be performed in the specific order or sequence shown, or requiring all illustrated operations to be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0137] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims may be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for training an environment representation neural network having multiple environment representation parameters, the environment representation neural network receiving input including observations characterizing the state of an environment interacting with an agent, and generating an internal representation of the state of the environment, wherein, The internal representation is used to control the agent in response to the observation, and wherein the method includes: Receive (i) a current observation input comprising a current observation characterizing the current state of the environment and (ii) a future observation input comprising a future observation characterizing a future state following the current state of the environment, wherein the observation includes: an image; or object position data captured by a distance or position sensor when the agent interacts with the environment; or sensor data from an actuator characterizing one or more of the robot's joint position, joint velocity, joint force, torque, acceleration, or global or relative posture of an object held by the robot; or sensor data from one or more sensors monitoring current, voltage, power, or temperature in a plant or service facility; or electronic signals representing the function of electronic or mechanical equipment; or analog versions of any of the foregoing. A future potential representation of the future state of the environment is generated from the future observation input by using a potential embedding neural network with multiple potential embedding network parameters and processing the future observation according to the current values ​​of the potential embedding network parameters to generate a future potential representation. The environment representation neural network is used to process the future observation input based on the current value of the environment representation parameters to generate a future internal representation of the future state of the environment; A predicted future internal representation is generated from the future potential representation, wherein the predicted future internal representation is a prediction of the future internal representation of the future state of the environment; The update of the current value of the potential embedded network parameters is determined based on the difference between the future internal representation and the predicted future internal representation; Using the environment representation neural network, the current observation input is processed according to the current value of the environment representation parameters to generate a current internal representation of the current state of the environment; Generate a predicted future potential representation from the current internal representation, wherein the predicted future potential representation is a prediction of the future potential representation of the future state of the environment; The objective function for evaluating the difference between the stated future potential representation and the predicted future potential representation; and The update of the current value of the environment representation parameter is determined based on the gradient of the determined objective function.

2. The method according to claim 1, wherein, The agent is controlled using a fixed, uniformly random action selection strategy.

3. The method according to any one of claims 1-2, further comprising: The current internal representation is processed using an action selection policy neural network with multiple policy parameters to generate a current action selection output specifying the action to be performed by the agent in the current state. The reinforcement learning loss is determined based on the current action selection output. as well as The update of the current value of the policy parameter is determined based on the reinforcement loss.

4. The method according to claim 3, further comprising: The update of the current value of the environment representation parameters is determined by backpropagating the gradient of the reinforcement learning loss into the environment representation neural network.

5. The method according to claim 1, wherein, The future state is k steps after the current state, where k is a predetermined positive integer.

6. The method according to claim 1, wherein: Generating the predicted future potential representation includes processing the feedforward prediction network input derived from the current internal representation using a feedforward prediction neural network to generate the predicted future potential representation, which is a prediction of the future potential representation of the future state of the environment; as well as The method further includes determining an update to the current value of the forward prediction network parameters based on calculating the gradient of the objective function relative to the forward prediction network parameters.

7. The method according to claim 1, wherein: Generating the predicted future internal representation includes processing the future latent representation using a backward prediction neural network with multiple backward prediction network parameters to generate the predicted future internal representation; and The method further includes determining an update to the current values ​​of the backward prediction network parameters.

8. The method according to claim 1, wherein, The current observation input also includes the preceding actions performed by the agent in the preceding state of the environment.

9. The method according to claim 6, wherein, The environment representation neural network includes a partial representation neural network configured to process (i) the current internal representation and (ii) the current action performed by the agent to generate a partial future internal representation of the future state of the environment.

10. The method according to claim 9, wherein, The forward prediction network input derived from the current internal representation is the partial future internal representation generated by the partial representation neural network.

11. The method of claim 6, further comprising: The calculated gradient is backpropagated into the environment representation neural network to determine an update to the current value of the environment representation parameters.

12. A system comprising one or more computers and one or more storage devices with storage instructions, wherein, when executed by said one or more computers, the instructions are operable to cause said one or more computers to perform operations according to a corresponding method according to any of the preceding claims.

13. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations of a corresponding method according to any one of claims 1-11.

Images