Action selection for reinforcement learning using manager and worker neural networks

Abstract

A system for selecting an action to be performed by an agent is provided, comprising: a worker neural network system configured to, at each of a plurality of time steps: receive a goal representation defining a goal to be accomplished as a result of an action performed by an agent in an environment, wherein the goal representation is based at least in part on an environment state of the time step and / or one or more previous time steps; and generate, based at least in part on the goal representation and the environment state of the time step, an action score for each action in a set of actions; and an action selection subsystem configured to, at each of the plurality of time steps: select, using the action scores, an action from the set of actions to be performed by the agent at the time step.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 201880013632.8, filed on February 19, 2018, entitled "Action Selection for Reinforcement Learning Using Neural Networks". Technical Field

[0002] This manual relates to reinforcement learning. Background Technology

[0003] In reinforcement learning systems, agents interact with the environment by performing actions selected by the reinforcement learning system in response to receiving observations that represent the current state of the environment.

[0004] Some reinforcement learning systems select the action that an agent should perform in response to a given observation, based on the output of a neural network.

[0005] A neural network is a machine learning model that uses one or more layers of non-linear units to predict the output in response to a received input. Some neural networks are deep neural networks, which include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output based on the received input, using the current values ​​of its corresponding parameter set. Summary of the Invention

[0006] This specification generally describes an action selection system that selects actions to be performed by a reinforcement learning agent that interacts with the environment.

[0007] In a first aspect, an action selection system is provided for selecting actions to be performed by an agent, wherein the agent interacts with an environment by performing actions from a predetermined set of actions. The system includes: a manager neural network subsystem configured to, at each of a plurality of time steps: receive an intermediate representation of the current state of the environment at the time step; map the intermediate representation to a latent representation of the current state in a latent state space; process the latent representation using a target recurrent neural network, wherein the target recurrent neural network is configured to receive the latent representation and process it according to the current hidden state of the target recurrent neural network to generate an initial target vector in the target space of the time step and update the internal state of the target recurrent neural network; and pool the initial target vector of the time step and one or more initial target vectors from previous time steps to generate a final target vector for the time step; and a worker neural network subsystem configured to: At each of the multiple time steps: receiving an intermediate representation of the current state of the environment at the time step, mapping the intermediate representation to the corresponding action embedding vector in the embedding space of each action in a predetermined action set, projecting the final target vector of the time step from the target space to the embedding space to generate a target embedding vector, and modulating the corresponding action embedding vector of each action by the target embedding vector to generate a corresponding action score for each action in the predetermined action set; and an action selection subsystem, wherein the action selection subsystem is configured to, at each of the multiple time steps: receiving an observation representing the current state of the environment at the time step, generating an intermediate representation from the observation, providing the intermediate representation as input to a manager neural network to generate a final target vector for the time step, providing the intermediate representation and the final target vector as input to a worker neural network to generate an action score, and using the action score to select an action to be performed by the agent in response to the observation from the predetermined action set.

[0008] In some implementations, selecting an action includes selecting the action with the highest action score.

[0009] In some implementations, generating intermediate representations from observations involves processing the observations using a convolutional neural network.

[0010] In some implementations, mapping the intermediate representation to a corresponding action embedding vector in the embedding space of each action in a predetermined action set includes: processing the intermediate representation using an action scoring recurrent neural network, wherein the action scoring recurrent neural network is configured to receive the intermediate representation and process it according to the current hidden state of the action scoring recurrent neural network to generate an action embedding vector and update the hidden state of the action scoring recurrent neural network.

[0011] In some implementations, mapping intermediate representations to potential representations of the current state involves processing intermediate representations using a feedforward neural network.

[0012] In some implementations, the target space has a higher dimension than the embedding space.

[0013] In some implementations, the dimension of the target space is at least ten times higher than the dimension of the embedding space.

[0014] In some implementations, the worker neural network subsystem has been trained to generate action scores that maximize the time-discounted combination of rewards, where each reward is a combination of an external reward received as a result of an agent performing a selected action and an intrinsic reward that depends on the target vector generated by the manager neural network subsystem.

[0015] In some implementations, the manager neural network subsystem has been trained to generate an initial target vector that induces action scores that encourage the agent to choose actions that move in favorable directions in the latent state space.

[0016] In some implementations, the target recurrent neural network is an extended long short-term memory (LSTM) neural network.

[0017] In a second aspect, a system for implementing an extended LSTM neural network is provided, wherein the extended LSTM neural network is configured to maintain an internal state divided into r substates, where r is an integer greater than 1, and wherein the extended LSTM neural network is configured to, at each time step in a time step sequence: receive a network input for the time step; select a substate from the r substates; and process the current value of the selected substate and the network input for the time step using the LSTM neural network to update the current value of the selected substate and generate a network output for the time step based on the current value of the LSTM network parameter set.

[0018] In some implementations, the extended LSTM neural network is also configured to, for each time step in the time step, pool the network output of the time step and pool the network output of the previous time step up to a predetermined number of times to generate the final network output of the time step.

[0019] In some implementations, pooling the network output includes summing the network outputs.

[0020] In some implementations, pooling network outputs includes averaging the network outputs.

[0021] In some implementations, pooling network outputs involves selecting the highest-performing network output.

[0022] In some implementations, time steps in the time step sequence are indexed from 1 at the first time step in the sequence to T at the last time step in the sequence, wherein each substate is assigned an index ranging from 1 to r, and wherein selecting a substate from r substates includes selecting a substate with an index equal to the index modulo r of the time step.

[0023] In some implementations, the LSTM neural network includes multiple LSTM layers.

[0024] In some implementations, using an LSTM neural network to process the current value of a selected substate and the network input at the time step to update the current value of the selected substate and to generate the network output at the time step based on the current value of the LSTM network parameter set includes: setting the internal state of the LSTM neural network to the current value of the selected substate so that the network input is processed at the time step.

[0025] In another aspect, a system is provided for selecting actions to be performed by an agent that interacts with an environment by performing actions from an action set. The system includes one or more computers and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to implement: a worker neural network system configured, at each of a plurality of time steps: receiving a target representation that defines a target to be accomplished as a result of an action performed by the agent in the environment, wherein the target representation is based at least in part on the environmental state of the time step and / or one or more previous time steps; and generating a corresponding action score for each action in the action set, based at least in part on the target representation and the environmental state of the time step; and an action selection subsystem configured, at each of the plurality of time steps: using the action score to select an action from the action set to be performed by the agent at that time step.

[0026] In another aspect, a method executed by one or more computers is provided for selecting an action to be performed by an agent, the agent interacting with an environment by performing actions from a set of actions, the method comprising, at each of a plurality of time steps: generating a target representation, the target representation defining a target to be accomplished as a result of an action performed by the agent in the environment, wherein the target representation is based at least in part on the environment state of the time step and / or one or more previous time steps; generating a corresponding action score for each action in the set of actions based at least in part on the target representation and the environment state of the time step; and selecting an action to be performed by the agent at the time step from the set of actions using the action score.

[0027] In another aspect, a non-transitory computer storage medium is provided for storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform operations for selecting actions to be performed by an agent, the agent interacting with an environment by performing actions from a set of actions, the operations comprising, at each of a plurality of time steps: generating a target representation, the target representation defining a target to be accomplished as a result of an action performed by the agent in the environment, wherein the target representation is based at least in part on the environment state of the time step and / or one or more previous time steps; generating a corresponding action score for each action in the set of actions, based at least in part on the target representation and the environment state of the time step; and selecting an action to be performed by the agent at the time step from the set of actions using the action score.

[0028] The above aspects can be implemented in any convenient form. For example, each aspect and implementation method can be implemented by a suitable computer program that can be carried on a suitable carrier medium, which can be a tangible carrier medium (e.g., a disk) or an intangible carrier medium (e.g., a communication signal). Each aspect can also be implemented using a suitable device that can take the form of a programmable computer running a computer program.

[0029] Specific embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages. The architecture of the action selection system described herein (including one or both of a decoupled manager-worker system architecture and an extended LSTM network) enables the system to be efficiently trained to perform long-term credit allocation (i.e., recognizing the influence of previous states and actions at many previous time steps on the received reward). Specifically, the architecture of the action selection system enables backpropagation of the system's gradients (according to a reinforcement learning algorithm) in response to the reward received at a given time step, thereby taking into account states and actions at many previous time steps when adjusting the system's parameters. Because the architecture of the action selection system facilitates long-term credit allocation, training the action selection system described herein can consume fewer computational resources (e.g., memory and computing power) than training other action selection systems with different architectures. Specifically, the action selection system described herein may require fewer training iterations to perform a task with a certain performance (e.g., the long-term time-discounted reward received by the agent while performing the task) than another system with a different architecture.

[0030] Furthermore, the manager system of the action selection system described in this specification generates directional goals rather than absolute goals for the worker system. Specifically, the worker system is trained to generate actions that cause the latent state representation of the manager system to move in a favorable direction in the latent state space, rather than being trained to explicitly move the latent state representation closer to an absolute goal in the latent state space. The worker system can generate actions that cause the latent state representation of the manager system to move in a specific direction in the latent state space more effectively than it can generate actions that move the latent state representation closer to an absolute goal in the latent state space. Therefore, by using directional goals instead of absolute goals, the action selection system described in this specification can achieve better performance (e.g., higher long-term time-discounted rewards received by the agent) and faster training (and thus consume fewer computational resources) than systems using absolute goals.

[0031] The extended Long Short-Term Memory (LSTM) neural network described in this specification can efficiently retain its internal memory over a large number of time steps while still being able to generate outputs at each time step and learn from each input. Therefore, the extended LSTM neural network is capable of generating sequential outputs, where a given output can depend on inputs much earlier in the input sequence. Thus, incorporating an extended LSTM network into the action selection system described in this specification can improve the performance of an agent's interaction with the environment based on the actions selected by the action selection system (e.g., by improving the agent's performance on a particular task, as reflected in the increased long-term time-discounted reward received by the agent when performing a particular task).

[0032] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0033] Figure 1 This is a diagram of an example action selection system.

[0034] Figure 2 This is a flowchart of an example process for selecting actions from a predetermined set of actions using an action selection system.

[0035] Figure 3 This is a flowchart of an example process for generating the final target vector of time steps using a manager system.

[0036] Figure 4 This is a flowchart of an example process for generating action scores for time steps using a worker system.

[0037] Figure 5 This is a flowchart of an example process for training an action selection system.

[0038] Figure 6 This is a flowchart of an example process for generating network output using an extended Long Short-Term Memory (LSTM) network.

[0039] The same reference numerals and names in the various figures denote the same elements. Detailed Implementation

[0040] Figure 1 An example action selection system 100 is shown. Action selection system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, wherein the systems, components, and techniques described below are implemented.

[0041] Action selection system 100 selects actions 144 to be performed by agent 142 interacting with environment 140 at each of multiple time steps. In order for agent 142 to interact with environment 140, system 100 receives observations 136 characterizing the current state of environment 140 and selects actions 144 to be performed by agent 142 in response to the received observations 136.

[0042] In some implementations, environment 140 is a simulated environment, and agent 142 is implemented as one or more computer programs that interact with the simulated environment by navigating through it. For example, the simulated environment may be a video game, and agent 142 may be a simulated user playing the video game by navigating through the simulated environment. As another example, the simulated environment may be a motion simulation environment, such as a driving simulator or a flight simulator, and agent 142 may be a simulated vehicle navigating through the motion simulation environment. In these cases, action 144 may be a control input controlling the simulated user or the simulated vehicle.

[0043] In some other implementations, environment 140 is a real-world environment, and agent 142 is a mechanical agent that interacts with the real-world environment. For example, agent 142 could be a robot that interacts with environment 140 to perform a specific task. As another example, agent 142 could be an autonomous or semi-autonomous vehicle navigating through the environment. In these cases, action 144 could be a control input that controls the robot or autonomous vehicle.

[0044] In some of these embodiments, observation 136 may be generated by or derived from the sensor of agent 142. For example, observation 136 may be an RGB image captured by the camera of agent 142. As another example, observation 136 may be derived from data captured from the laser sensor of agent 142. As yet another example, observation 136 may be a hyperspectral image captured by the hyperspectral sensor of agent 142.

[0045] At each time step, system 100 receives an external reward 138 based on the current state of environment 140 and the action 144 of agent 142. Typically, the external reward is a numerical value reflecting the effect of performing action 144 on agent 142's completion of a specific task. For example, system 100 may receive the external reward 138 for a given time step based on progress toward achieving one or more goals for agent 142. For example, the agent's goal might be navigating to a target location within environment 140.

[0046] System 100 includes an encoding neural network 120 configured to receive observations 136 at a time step, process the observations based on current values ​​of the encoding neural network parameter set, and generate an intermediate representation 118 of the observations 136 at the time step as output. The intermediate representation 118 may be an ordered set of numerical values ​​(e.g., a vector or matrix) that represents the observations 136 at the time step in an intermediate state space (e.g., the Euclidean space to which the intermediate representation 118 belongs). Specifically, when the observations 136 are images, the encoding neural network 120 may be a convolutional neural network.

[0047] System 100 includes a manager system 102 and a worker system 116. At each time step, the manager system 102 is configured to generate an abstract goal (i.e., an initial goal vector 110). At each time step, the worker system 116 is configured to generate an action score 132, which causes an agent 142 to perform actions aimed at implementing the abstract goal generated by the manager system 102.

[0048] System 100 provides the intermediate representation 118 of the time step as input to manager system 102, which is configured to receive the intermediate representation 118 of the time step as input, process the intermediate representation according to the current value of the manager system parameter set, and generate the final target vector 114 of the time step as output.

[0049] The management system includes a latent representation neural network 104 configured to receive an intermediate representation 118 at a time step as input, process the intermediate representation based on the current values ​​of the latent representation neural network parameter set, and generate a latent representation 106 at the time step as output. The latent representation 106 is an ordered set of numerical values ​​(e.g., vectors or matrices) that represents the intermediate representation 118 in a latent state space (e.g., the Euclidean space to which latent representation 106 belongs) and (indirectly) represents the observation 136 at the time step. In some embodiments, the latent representation neural network 104 is a feedforward neural network (e.g., a multilayer perceptron). Therefore, the management system can apply operations, for example, using a latent representation neural network (...). The intermediate representation is mapped to a latent representation of the current state in the latent state space, so that the manager system can then use the latent state space representation to define the objective. The objective can be represented by an objective vector in what can be defined as the objective space. In some implementations, the latent state space can be learned, as described later. Using such a latent representation can facilitate setting higher-level objectives and thus can promote the learning of complex tasks involving sub-objectives and / or with delayed rewards.

[0050] The management system 102 includes a target recurrent neural network 108 configured to process the latent representation 106 of each time step and the internal state of the target recurrent neural network 108 at each time step. The target recurrent neural network 108 updates its internal state and generates an initial target vector 110 for each time step as its output. The initial target vector 110 for each time step is an ordered set of values ​​logically arranged as vectors. The target recurrent neural network 108 can be implemented as any suitable recurrent neural network model. In some embodiments, the target recurrent neural network 108 is an LSTM network. In some embodiments, the target recurrent neural network 108 is an extended LSTM network. An extended LSTM network is an LSTM network that generates network outputs for each time step but only processes and updates a subset of its internal state. (Reference) Figure 6 An example procedure for generating network output (e.g., initial target vector 110) using an extended LSTM network is described.

[0051] The management system 102 includes a target pooling engine 112 that pools (i.e., combines) the initial target vector 110 of a time step with one or more initial target vectors from previous time steps to generate a final target vector 114 for the time step. The final target vector 114 of the time step is an ordered set of values ​​logically arranged into a vector. Due to the pooling operations performed by the target pooling engine 112, the final target vector 114 changes smoothly between time steps. (Reference) Figure 3 An example process for generating the final target vector of time steps using a manager system is described.

[0052] System 100 provides intermediate representations 118 of time steps and final target vectors 114 of time steps (as generated by manager system 102) as inputs to worker system 116.

[0053] Worker system 116 is configured to process inputs based on the current values ​​of the worker system parameter set to generate an action score 132 as output. The system uses the action score 132 to select the action 144 to be performed by agent 142 for a given time step.

[0054] Action score 132 includes a separate score for each action in a predetermined set of actions (i.e., actions that agent 142 can take). In some embodiments, system 100 selects action 144 to be performed by agent 142 as the action with the highest corresponding score based on action score 132. In some other embodiments, the system processes action score 132 using a softmax function to determine the probability of each action in the predetermined set of actions. In these embodiments, the system can select action 144 to be performed by agent by sampling from the probability distribution in the predetermined set of actions determined based on action score 132.

[0055] The worker system 116 includes an action-scoring recurrent neural network 122 configured to process intermediate representations 118 at time steps and the internal state of the action-scoring recurrent neural network 122 at each time step. The action-scoring recurrent neural network 122 updates its internal state and generates a set of action embedding vectors 124 for each time step as output. The action embedding vector set 124 includes the embedding vector for each action in a predetermined action set. An action embedding vector is an ordered set of values ​​arranged as vectors in an embedding space (by definition). Therefore, in embodiments performed by the action-scoring recurrent neural network of the worker system, intermediate representations can be mapped from the worker system to the embedding space. The action-scoring recurrent neural network 122 can be implemented as any suitable recurrent neural network model. In some embodiments, the action-scoring recurrent neural network 122 is implemented as an LSTM network. In some embodiments, the action-scoring recurrent neural network 122 is implemented as an extended LSTM.

[0056] The worker system 116 includes a projection engine 126 configured to receive a final target vector 114 for a time step generated by the manager system 102, process the final target vector 114 based on current values ​​of the projection engine parameter set, and generate a target embedding vector 130 in the embedding space as output. The target embedding vector 130 is an ordered set of values ​​arranged into vectors. Typically, the target embedding vector 130 has a lower dimension than the final target vector 114. In some implementations, the dimension of the target embedding vector 130 may be ten times or more lower than the dimension of the final target vector 114 (i.e., one order or order of magnitude lower or more).

[0057] The worker system 116 includes a modulation engine 128 configured to receive an action embedding vector 124 and a target embedding vector 130 at a time step as input and generate an action score 132 as output. Therefore, the action embedding vector is modulated by the target embedding vector, for example, by weighting the action embedding vector using the target embedding vector, i.e., modifying it according to the value of the target embedding vector. (Reference) Figure 4An example process for generating action scores for time steps using a worker system is described.

[0058] The system selects the action 144 to be performed by the agent 142 based on the action score 132 generated by the worker system 116. (Reference) Figure 2 An example process for selecting actions from a predetermined set of actions using an action selection system is described.

[0059] System 100 includes a training engine 134 configured to train system 100 according to a reinforcement learning training method by adjusting the parameters of system 100, including the parameters of manager system 102, worker system 116, and encoded neural network 120.

[0060] Typically, training engine 134 trains worker system 116 to generate action scores 132 that maximize a long-term time discount combination of the extrinsic reward 138 received by agent 142 as a result of performing selected action 144 and the intrinsic reward determined by training engine 134. Training engine 134 determines the intrinsic reward based on how well the action scores 132 generated by worker system 116 cause latent representation 106 to move in an orientation within the latent state space given by initial target vector 110 generated by manager system 102. Training engine 134 trains manager system 102 to generate initial target vector 110 that induces action scores 132 that encourage the selection of actions 144 that cause latent representation 106 to move in favorable orientations within the latent state space (e.g., orientations that induce receiving a larger long-term time discount extrinsic reward 138). Reference Figure 5 An example process for training an action selection system is described.

[0061] Figure 2 This is a flowchart illustrating an example process of selecting actions from a predetermined set of actions using an action selection system. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to this specification, such as... Figure 1 The action selection system 100 can execute process 200.

[0062] The system receives observations characterizing the current state of the environment (202). In some embodiments, the environment is a simulated environment, and the agent is implemented as one or more computer programs that interact with the simulated environment through navigation. In some other embodiments, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment. In some embodiments, observations may be generated by or derived from the agent's sensors.

[0063] The system generates intermediate representations of the observations at each time step (204). Specifically, the system provides the observations as input to an encoding neural network configured to process the observations based on the current values ​​of the encoding neural network's parameter set, and generates intermediate representations of the observations at each time step as output. The intermediate representation is an ordered set (e.g., a vector or matrix) representing the numerical values ​​of the observations at each time step in an intermediate state space. In particular, when the observations are images, the encoding neural network can be a convolutional neural network. The values ​​of the encoding neural network's parameter set are determined during training of the system by the training engine.

[0064] The system provides intermediate representations of time steps to the management system, and the management system processes these intermediate representations to generate the final target vector of the time steps as output (206). See below for reference. Figure 3 Describe an example process for generating the final target vector of time steps using a manager system.

[0065] The system provides intermediate representations of time steps and the final target vector of the time steps (as generated by the manager system) as inputs to the worker system. The worker system processes the inputs of the time steps to generate action scores for the time steps as outputs (208). Reference Figure 4 Describe an example process for generating action scores for time steps using a worker system.

[0066] The system selects actions from a predetermined action set based on action scores generated by the worker system (210). The action scores include a separate score for each action in the predetermined action set. In some implementations, the system selects the action to be performed by the agent based on the action score as the action with the highest corresponding score. In other implementations, the system processes the action scores using a softmax function to determine the probability of each action in the predetermined action set. In these implementations, the system can select the action to be performed by the agent by sampling from the probability distribution of the predetermined action set determined based on the action scores.

[0067] Figure 3 This is a flowchart of an example process for generating the final target vector of time steps using a management system. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a management system appropriately programmed according to this specification, such as... Figure 1 The administrator system 102 can execute process 300.

[0068] The system receives an intermediate representation of the current state of the environment at a time step (302). Typically, the intermediate representation of the current state of the environment is generated by an encoded neural network.

[0069] The system generates a latent representation for a time step by providing an intermediate representation of that time step as input to the latent representation neural network (304). The latent representation neural network is configured to process the intermediate representation of the time step based on the current values ​​of the latent representation neural network parameter set to generate a latent representation of the time step as output. In some implementations, the latent representation neural network is a feedforward neural network (e.g., a multilayer perceptron). The latent representation of a time step is an ordered set of numerical values ​​(e.g., vectors or matrices) that represents the intermediate representation and (indirectly) the observations at the time step in the latent state space. The values ​​of the latent neural network parameter set are determined during training of the system by the training engine.

[0070] The system generates an initial target vector (306) for each time step. Specifically, the system provides the target recurrent neural network with the latent representation of the time step generated by the latent representation neural network. The target recurrent neural network is configured to process the latent representation of the time step and the internal state of the target recurrent neural network at each time step. The target recurrent neural network updates the internal state of the target recurrent neural network and generates the initial target vector for the time step as its output, i.e.:

[0071]

[0072] in It is the updated internal state of the target recurrent neural network (i.e., the internal state of the target recurrent neural network for the next time step). It is the internal state of the target recurrent neural network at each time step. It is a potential representation of time steps. It is the initial target vector at time step, and This refers to the operations performed by the target recurrent neural network (RNN) based on the current values ​​of its parameter set. The initial target vector at each time step is a logically ordered set of values ​​arranged as vectors. In some implementations, the system normalizes the initial target vector at each time step to have a unit length (e.g., relative to the Euclidean norm). The target RNN can be implemented as any suitable recurrent neural network model. In some implementations, the target RNN is an LSTM network. In some implementations, the target RNN is an extended LSTM network. The values ​​of the target RNN parameter set are determined during training by the training engine. (See reference...) Figure 6 Describe an example procedure for generating network outputs (e.g., initial target vectors) using an expanded LSTM network.

[0073] The system generates the final target vector for a time step by pooling the initial target vector generated by the target recurrent neural network for the time step and the initial target vector generated by the target recurrent neural network for one or more previous time steps (308). In some implementations, the system pools the initial target vectors (i.e., the initial target vectors generated for the current time step and one or more previous time steps) by summing them. However, the target vectors can be combined in many other ways. The final target vector for a time step is an ordered set of values ​​arranged into vectors. Due to the pooling operation, the final target vector changes smoothly between time steps.

[0074] Figure 4 This is a flowchart of an example process for generating motion scores for time steps using a worker system. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a worker system appropriately programmed according to this specification, such as... Figure 1 The worker system 116 can execute process 400.

[0075] The system receives an intermediate representation of the current state of the environment and the final target vector of the time step (402). Typically, the intermediate representation of the current state of the environment is generated by an encoding neural network, and the final target vector of the time step is generated by the manager system.

[0076] The system generates an action embedding vector (404) for each time step. Specifically, the system provides the intermediate representation of the time step generated by the encoding neural network to the action scoring recurrent neural network. The action scoring recurrent neural network is configured to process the intermediate representation of the time step and the internal state of the action scoring recurrent neural network for each time step. The action scoring recurrent neural network updates its internal state and generates the action embedding vector for each time step as its output, i.e.:

[0077]

[0078] in It is the updated internal state of the action scoring recurrent neural network (i.e., the internal state of the action scoring recurrent neural network at the next time step). It is the internal state of the action-scoring recurrent neural network at each time step. It is an intermediate representation of the time step. It is a set of action embedding vectors for each time step arranged in a matrix (where each row corresponds to a different action embedding vector), and This refers to the operations performed by the action scoring recurrent neural network based on the current values ​​of its parameter set. The action scoring recurrent neural network can be implemented as any suitable recurrent neural network model. For example, it can be implemented as an LSTM network or an extended LSTM network.

[0079] The system generates a target embedding vector (406) for the time step based on the final target vector of the time step. Specifically, the system provides the final target vector of the time step to a projection engine configured to receive the final target vector of the time step, process the final target vector of the time step according to the current values ​​of the projection engine parameter set, and generate the target embedding vector of the time step as output. Typically, the target embedding vector has a lower dimension than the final target vector. In some implementations, the dimension of the target embedding vector may be ten times or more lower than the dimension of the final target vector (i.e., one order of magnitude or more lower). In some implementations, the operation of the projection engine can be given as follows:

[0080]

[0081] The current time step is t. For the target embedding vector at time step, Let φ be the final target vector at time step φ, and let φ be the projection matrix (i.e., having the same properties as the vector). The dimensions have the same number of columns and have the same dimensions as the target embedding vector. The final target vector for a time step is determined by summing the initial target vectors generated for the current time step and the previous c time steps (e.g., by the manager system in 308), where c is a positive integer.

[0082] The system generates an action score (408) for each time step. Specifically, the system provides the target embedding vector and the action embedding vector for each time step as input to a modulation engine configured to combine the inputs to generate the action score for each time step as output. In some implementations, the modulation engine is configured to combine the target embedding vector and the action embedding vector for each time step via matrix-vector multiplication, i.e.:

[0083]

[0084] in It is a matrix, where each row of the matrix is ​​the action embedding vector for each time step. It is the target embedding vector at time step, and It is the action score vector at the time step.

[0085] Figure 5This is a flowchart of an example process for training an action selection system. For convenience, process 500 will be described as being executed by an engine comprising one or more computers located in one or more locations. For example, a training engine appropriately programmed according to this specification, such as... Figure 1 The training engine 134 can execute process 500.

[0086] The engine receives an external reward for a time step (502). The received external reward is based on the current state of the environment and the agent's action at that time step. Typically, the external reward is a numerical value reflecting the effect of performing the action on the agent's completion of a specific task. In some cases, the engine may receive an external reward for a given time step based on the agent's progress toward achieving one or more goals. For example, the agent's goal might be to navigate to a target location in the environment.

[0087] The engine determines the intrinsic reward for time step t (504). Typically, the engine determines the intrinsic reward based on the extent to which the action scores generated by the worker system cause the latent representation (e.g., generated by the manager system in 304) to move in a direction within the latent state space given by the initial target vector generated by the manager system (e.g., in 306). For example, the engine can determine the intrinsic reward for time step t using the following equation:

[0088]

[0089] in It is the intrinsic reward at time step t, where c is a positive integer, and d is the intrinsic reward at time step t. cos It is the cosine similarity between two vectors (i.e., ), s t It is the latent representation of time step t, s t-i It is the potential representation of time step ti, and g t-i Let be the initial target vector at time step ti. Therefore, the intrinsic reward can depend on the difference in direction between the target vector at time step and the vector representing the change in the potential representation from that time step to the next time step.

[0090] The engine adjusts the parameters of the worker system based on extrinsic and intrinsic rewards (506). Specifically, the engine adjusts the parameters of the modulation engine, projection engine, action scoring recurrent neural network, and encoding neural network so that the worker system generates an action score that maximizes a long-term time-discounted combination of the extrinsic reward received as a result of the agent performing the selected action and the intrinsic reward determined by the training engine. For example, the long-term time-discounted combination can be given by the following equation:

[0091]

[0092] Where γ and It is a discount factor between 0 and 1 (which may usually be different), r t+k+1 The external reward received at time step t+k+1 It is the intrinsic reward received at time step t+k+1. This is a constant that modulates the influence of intrinsic rewards and can be determined by any suitable method (e.g., arbitrarily or through cross-validation). The engine can train the parameters of the worker system using any suitable reinforcement learning method. For example, the engine can use a dominant action evaluation reinforcement learning method to train the parameters of the worker system (e.g., where the value function estimates of intrinsic and extrinsic rewards are generated by a neural network, such as a multilayer perceptron, which receives the internal states of an action-scoring recurrent neural network as input). In some implementations, the engine trains the worker system independently of the manager system (i.e., the engine does not backpropagate gradients from the worker system and the manager system, making the objective not merely an intrinsic variable of a single system). This is useful because it facilitates the independent development of the manager system's objectives based on extrinsic rewards, while enabling the worker system to achieve these objectives based on intrinsic rewards.

[0093] The engine updates the parameters of the manager system based on external rewards (508). Specifically, the engine adjusts the parameters of the target recurrent neural network, the latent representation neural network, and the encoding neural network. Adjusting the parameters of the manager system causes it to generate an initial target vector that incentivizes actions that move the latent representation of the manager system in a favorable direction in the latent state space. A direction in the latent state space is considered favorable if, for example, moving the latent representation in a direction results in receiving a larger long-term time discount external reward. In some implementations, the parameters of the manager system can be updated via gradient ascent, where the gradient at time step t+c is given by:

[0094]

[0095] in This refers to the parameters of the management system. This refers to parameters The derivative of , where c is a positive integer, is called the range (horizon) defining the time resolution of the management system, d cos It is the cosine similarity between two vectors (as described above), s t+c It is the latent representation at time t+c, s t It is the latent representation at time t. It is the initial target vector at time step t (where the notation explicitly indicates the parameters). (dependence) This is known as the advantage function of the manager system, and Based on parameters Observation of x at time step t t The calculated value function estimate (i.e., the estimate of the value function from observations of the environment by x) t The expected long-term time-discounted external reward is given at the start of a given time step. A neural network (e.g., a multilayer perceptron) that receives the internal state of the target recurrent neural network as input and is trained (e.g., via gradient backpropagation) to generate an accurate estimate of the expected long-term time-discounted external reward is used to determine the value function estimate. In these implementations, the engine backpropagates gradients to adjust the parameters of the manager system after every c time steps.

[0096] Figure 6 This is a flowchart of an example process for generating network output using an extended LSTM network. For convenience, process 600 will be described as being performed by a neural network comprising one or more computers located at one or more locations. For example, a neural network appropriately programmed according to this specification, such as... Figure 1 The target recurrent neural network 108 can execute process 600.

[0097] Typically, an extended LSTM network is an LSTM network consisting of a sequence of one or more LSTM layers. Each LSTM layer maintains an internal state that is divided into multiple disjoint substates. The internal states of an LSTM layer are ordered sets of numerical values ​​(e.g., represented as vectors or matrices). The substates of an LSTM layer are subsets of the ordered sets of numerical values ​​of the internal states.

[0098] The network receives network inputs for each time step (602). Typically, the network receives the corresponding network input for each time step sequentially. For example, the network input for a time step could be a latent representation of that time step generated by a latent representation neural network.

[0099] Each LSTM network layer selects a substate for a time step (604). Time steps in the time step sequence can be indexed from 1 (the first time step in the sequence) to T (the last time step in the sequence), and for a specific LSTM network layer, each substate of the LSTM network layer can be assigned an index ranging from 1 to r. A specific LSTM network layer can select a substate for a time step with a substate index equal to the index of the time step modulo r. For example, if each time step is assigned an index from 1 to 10, and each substate is assigned an index from 1 to 3, then the substate selected for the time step assigned index 1 will be the substate assigned index 1, and the substate selected for the time step assigned index 5 will be the substate assigned index 2.

[0100] The current value of a substate can include the output values ​​of the nodes in the (multiple) LSTM network layers indexed by the substate. For each LSTM network layer, the LSTM network layer processes the current value of the substate selected for the time step and the input of the time step according to the LSTM network layer parameter set to update the current value of the substate selected for the time step and generate the output of the time step (606). The first LSTM network layer in the sequence receives the network input of the time step. Each subsequent LSTM network layer receives the output of the previous LSTM network layer in the sequence of time steps as the input of the time step. Each LSTM network layer can perform the following operations:

[0101]

[0102] Where t is the current time step, r is the number of sub-states in the LSTM network layer, and % refers to the modulo operation. This refers to the updated value of the sub-state of the LSTM network layer indexed by t%r (i.e., the value of the sub-state indexed by t%r at time step t+1). This refers to the value of the sub-state of the LSTM network layer indexed by t%r at time step t. This refers to the output of the LSTM network layer at each time step. This refers to the input of the LSTM network layer at each time step. This refers to the parameter set of the LSTM network layer, and LSTM refers to the LSTM network layer based on the parameter set of the LSTM network layer. The operations performed. Typically, each LSTM network layer processes the input LSTM network layer parameter set. It is independent of the sub-state chosen for the time step. Different LSTM network layers can have different LSTM network layer parameter values. The complete state of the extended LSTM network can be specified as... Furthermore, each of the r substates or substate groups can be considered a separate "core" of the extended LSTM network.

[0103] The network determines the network output for a time step (608). In some embodiments, the network determines the network output for a time step as the output of the final LSTM network layer for that time step. In other embodiments, the network determines the network output for a time step by pooling (i.e., combining) the output of the final LSTM network layer for that time step with up to a predetermined number of outputs from previous time steps. The network can pool the final LSTM network layer output by summing them, by averaging them, by selecting the final LSTM network layer output with the highest norm, or by any other suitable pooling method. This is beneficial for the expanded LSTM network to retain memory over a longer time period.

[0104] This specification uses the term "configured" in relation to system and computer program components. For a system of one or more computers configured to perform specific operations or actions, this means that software, firmware, hardware, or a combination thereof are installed on the system, which in operation causes the system to perform operations or actions. For one or more computer programs configured to perform specific operations or actions, this means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform operations or actions.

[0105] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware including the structures disclosed in this specification and their equivalents, or in a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0106] The term "data processing apparatus" refers to data processing hardware and includes all types of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0107] A computer program, also referred to or described as a program, software, software application, application, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program can, but does not have to, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as multiple coordinated files (e.g., a file storing one or more modules, subroutines, or code sections). A computer program can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected via a data communication network.

[0108] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same computer or multiple computers.

[0109] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry (such as FPGAs or ASICs) or a combination of special-purpose logic circuitry and one or more programmable computers.

[0110] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to one or more mass storage devices for storing data. However, a computer does not necessarily have to have such devices. Furthermore, a computer can be embedded in another device, to name just a few, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive).

[0111] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0112] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in return.

[0113] The data processing apparatus for implementing machine learning models may also include, for example, a dedicated hardware accelerator unit for processing the common and computationally intensive portions of machine learning training or production, namely, inference, workloads.

[0114] Machine learning frameworks can be used to implement and deploy machine learning models, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0115] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, or middleware components, or frontend components, or any combination of one or more such backend, middleware, or frontend components. The backend components may be, for example, data servers; the middleware components may be, for example, application servers; and the frontend components may be, for example, graphical user interfaces, web browsers, or client computers through which users can interact with embodiments of the subject matter described in this specification. The components of the system can be interconnected via any form of digital data communication medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0116] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by computer programs running on separate computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to a user device, for purposes such as displaying data to a user interacting with the device as a client and receiving user input from the user. Data generated on the user device, such as the result of user interaction, may be received at the server from the device.

[0117] While this specification contains numerous specific details of implementation, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from a claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0118] Similarly, although operations are depicted in a specific order in the accompanying drawings and described in the claims, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order, or to perform all of the shown operations to achieve the desired effect. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0119] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A system for selecting actions to be performed by an agent, the agent interacting with an environment by performing actions from a set of actions, the system comprising one or more computers and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform: An encoded neural network system having an encoded neural network parameter set, the encoded neural network system being configured as follows: Receive observations characterizing the current state of the environment, wherein the observations include one or more of the following: data generated by a camera sensor, data generated by a laser sensor, or data generated by a hyperspectral sensor; processing the observation in accordance with the current values of the sets of encoded neural network parameters to generate an intermediate representation of the observation; and Output the intermediate representation. A worker neural network system, configured to operate at each of multiple time steps: Receive the intermediate representation; Receive a target representation, the target representation defining a goal to be accomplished as a result of an action performed by the agent in the environment, wherein the target representation is based at least in part on the environment state of the time step and / or one or more previous time steps; and At least in part based on the intermediate representation, the target representation, and the environmental state at the time step, a corresponding action score is generated for each action in the action set; and The action selection subsystem is configured to, at each of the plurality of time steps: The action score is used to select the action to be performed by the agent at the time step from the action set.

2. The system of claim 1, wherein the encoded neural network comprises a convolutional neural network.

3. The system of claim 1, wherein the worker neural network system includes a projection engine having a projection engine parameter set, the projection engine being configured to: receiving the target representation; and The target representation is processed based on the current values ​​of the projection engine parameter set to generate a target embedding vector; and Output the target embedding vector. The generation of the corresponding action score for each action in the action set is at least partially based on the target embedding vector.

4. The system of claim 3, wherein, The worker neural network system is configured to map the intermediate representation to the corresponding action embedding vector for each action in the action set.

5. The system of claim 4, wherein the worker's nervous system includes a modulation engine, the modulation engine being configured to: Receive the corresponding action embedding vector; Receive the target embedding vector; Based on the corresponding action embedding vector and the target embedding vector for each action in the action set, a corresponding action score is generated for each action.

6. The system of claim 5, wherein the corresponding action score for generating the action includes: Modify the action embedding vector of the action based on the target embedding vector.

7. The system of claim 5, wherein the corresponding action score for generating the action includes: Combine the corresponding action embedding vector and the target embedding vector.

8. The system of claim 1, further comprising a manager neural network system configured to: at each of the plurality of time steps: Generate a latent representation of the current state of the environment at the given time step in the latent space; and The final target representation for the time step is generated using at least a potential representation of the current state of the environment at that time step.

9. The system of claim 8, wherein the manager neural network system is configured to receive an intermediate representation of an observation characterizing the current state of the environment from an encoding neural network, and wherein the generation of the latent representation is at least partially based on the intermediate representation.

10. The system of claim 9, wherein generating the latent representation includes processing the intermediate representation using a latent representation neural network.

11. The system of claim 10, wherein the latent representation neural network comprises a feedforward neural network.

12. The system of claim 8, wherein at each of the plurality of time steps, generating a final target representation for that time step using at least a potential representation of the current state of the environment at that time step comprises: An initial target representation for the time step is generated, at least in part, based on a potential representation of the current state of the environment at the time step. and The initial target representation of the time step is combined with the initial target representation of one or more previous time steps to generate the final target representation of the time step.

13. The system of claim 12, wherein, At each of the plurality of time steps, generating the initial target representation for that time step includes: The latent representation of the time step is processed using a target neural network having a target neural network parameter set, wherein the target neural network is configured to receive the latent representation and process the latent representation based on the current values ​​of the target neural network parameter set.

14. The system of claim 1, wherein at each of the plurality of time steps, selecting an action includes selecting the action with the highest action score.

15. The system of claim 1, wherein, The worker neural network system has been trained to generate action scores that maximize a time-discounted combination of rewards, where each reward is a combination of an external reward received as a result of the agent performing a selected action and an internal reward that depends on the target representation.

16. The system of claim 8, wherein, The manager's neural network system has been trained to generate a target representation that induces an action score that encourages the selection of actions that increase the external reward received as a result of the agent performing the selected action.

17. A method executed by one or more computers for selecting an action to be performed by an agent, the agent interacting with an environment by performing actions from a set of actions, the method comprising, at each of a plurality of time steps: Receive observations characterizing the current state of the environment, wherein the observations include one or more of the following: data generated by a camera sensor, data generated by a laser sensor, or data generated by a hyperspectral sensor; The observation is processed based on the current values ​​of the parameter set of the encoding neural network to generate an intermediate representation of the observation; and Output the intermediate representation; Generate a target representation, which defines a goal to be achieved as a result of an action performed by an agent in the environment, wherein the target representation is based at least in part on the environment state of the time step and / or one or more previous time steps; At least in part based on the intermediate representation, the target representation, and the environmental state at the time step, a corresponding action score is generated for each action in the action set; and The action score is used to select the action to be performed by the agent at the time step from the action set.

18. A non-transitory computer storage medium storing one or more instructions, said instructions, when executed by one or more computers, causing said one or more computers to perform operations for selecting actions to be performed by an agent, said agent interacting with an environment by performing actions from a set of actions, said operations including, at each of a plurality of time steps: Receive observations characterizing the current state of the environment, wherein the observations include one or more of the following: data generated by a camera sensor, data generated by a laser sensor, or data generated by a hyperspectral sensor; The observation is processed based on the current values ​​of the parameter set of the encoding neural network to generate an intermediate representation of the observation; and Output the intermediate representation. Generate a target representation, which defines a goal to be achieved as a result of an action performed by an agent in the environment, wherein the target representation is based at least in part on the environment state of the time step and / or one or more previous time steps; At least in part based on the intermediate representation, the target representation, and the environmental state at the time step, a corresponding action score is generated for each action in the action set; and The action score is used to select the action to be performed by the agent at the time step from the action set.

Images