Action Selection for Reinforcement Learning Using Neural Networks

The action selection system with a manager-worker neural network structure and extended LSTM network addresses long-term credit assignment issues, improving reinforcement learning performance by optimizing action selection for long-term rewards.

JP7587626B2Active Publication Date: 2024-11-20ジーディーエム·ホールディング·エルエルシー
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023060824
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2017-02-24
Filing Date
2023-04-04
Publication Date
2024-11-20
Estimated Expiration
2038-02-19

AI Technical Summary

Technical Problem

Reinforcement learning systems face challenges in effectively selecting actions that consider long-term rewards and credit assignment over multiple time steps, leading to inefficient resource consumption and suboptimal performance.

Method used

An action selection system utilizing a manager-worker neural network structure, including an extended LSTM network, processes latent representations to generate directional goals, allowing for long-term credit assignment and efficient action selection, reducing computational resources and improving performance by focusing on time-discounted rewards.

Benefits of technology

The system enhances performance by efficiently selecting actions that maximize long-term rewards, reducing the need for repetitive training and minimizing resource consumption, while maintaining effective interaction with environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007587626000021
    Figure 0007587626000021
  • Figure 0007587626000022
    Figure 0007587626000022
  • Figure 0007587626000023
    Figure 0007587626000023
Patent Text Reader

Abstract

To provide methods, systems, and devices, including computer programs encoded on a computer storage medium, for a system configured to select actions to be performed by an agent that interacts with an environment.SOLUTION: The system includes a manager neural network subsystem and a worker neural network subsystem. The manager subsystem is configured to, at each of multiple time steps, generate a final goal vector for the time step. The worker subsystem is configured to, at each of multiple time steps, use the final goal vector generated by the manager subsystem to generate a respective action score for each action in a predetermined set of actions.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This specification relates to reinforcement learning. [Background technology]

[0002] In a reinforcement learning system, an agent interacts with an environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.

[0003] Some reinforcement learning systems select an action to be performed by an agent in response to receiving a given observation according to the output of a neural network.

[0004] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input for the next layer in the network, i.e. the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. Summary of the Invention [Means for solving the problem]

[0005] This document generally describes an action selection system that selects actions to be performed by a reinforcement learning agent interacting with an environment.

[0006] In a first aspect, there is provided an action selection system for selecting an action to be performed by an agent that interacts with an environment by performing actions from a predetermined action set, the system comprising the steps of receiving, at each of a plurality of time steps, an intermediate representation of a current state of the environment at the time step, mapping the intermediate representation to a latent representation of the current state in a latent state space, and performing a goal regression based on the intermediate representation.a manager neural network subsystem configured to process the latent representations using a goal regression neural network (GRN) configured to receive the latent representations and process the latent representations according to a current hidden state of the goal regression neural network to generate an initial goal vector in a goal space for the time step and update an internal state of the goal regression neural network; and pooling the initial goal vector for the time step and the initial goal vectors for one or more previous time steps to generate a final goal vector for the time step. At each of a plurality of time steps, receiving an intermediate representation of a current state of the environment at the time step, mapping the intermediate representation to a respective action embedding vector in the embedding space for each action of a predetermined set of actions, and updating an internal state of the goal regression neural network. a worker neural network subsystem configured to project a final goal vector from the target space into the embedding space to generate a goal embedding vector and adjust a respective action embedding vector for each action with the goal embedding vector to generate a respective action score for each action in the predetermined set of actions; and a behavior selection subsystem configured to receive, at each of a plurality of time steps, observations characterizing a current state of the environment at the time step, generate an intermediate representation from the observations, provide the intermediate representation as input to a manager neural network to generate a final goal vector for the time step, provide the intermediate representation and the final goal vector as input to the worker neural network to generate an action score, and use the action score to select an action from the predetermined set of actions to be performed by the agent in response to the observation.

[0007] In some implementations, selecting the behavior includes selecting the behavior having the highest behavior score.

[0008] In some implementations, generating the intermediate representation from the observations includes processing the observations using a convolutional neural network.

[0009] In some implementations, mapping the intermediate representations to respective behavior embedding vectors in the embedding space for each behavior of the predetermined set of behaviors includes processing the intermediate representations using a behavior score regression neural network, where the behavior score regression neural network is configured to receive the intermediate representations and process the intermediate representations according to a current hidden state of the behavior score regression neural network to generate behavior embedding vectors and update the hidden states of the behavior score regression neural network.

[0010] In some implementations, mapping the intermediate representation to a latent representation of the current state includes processing the intermediate representation using a feedforward neural network.

[0011] In some implementations, the target space has a greater dimensionality than the embedding space.

[0012] In some implementations, the dimensionality of the target space is at least 10 times larger than the dimensionality of the embedding space.

[0013] In some implementations, the worker neural network subsystem is trained to generate an action score that maximizes a combination of time-discounted rewards, each of which is a combination of an extrinsic reward received as a result of the agent performing a selected action, and an intrinsic reward that depends on a goal vector generated by the manager neural network subsystem.

[0014] In some implementations, the manager neural network subsystem is trained to generate initial goal vectors that result in action scores that facilitate the selection of actions that move the agent in advantageous directions in the latent state space.

[0015] In some implementations, the goal regression neural network is an extended long short-term memory (LSTM) neural network.

[0016] In a second aspect, a system is provided for implementing an extended LSTM neural network, the extended LSTM neural network being configured to maintain an internal state divided into r substates, where r is an integer greater than 1, and configured, at each time step in a sequence of time steps, to receive a network input for the time step, select a substate from the r substates, process a current value of the selected substate for the time step and a current value of the network input using the LSTM neural network to update the current value of the selected substate, and generate a network output for the time step in accordance with current values ​​of a set of LSTM network parameters.

[0017] In some implementations, the extended LSTM neural network is further configured, for each time step, to pool the network output for the time step and up to a predetermined number of network outputs of previous time steps to generate a final network output for the time step.

[0018] In some implementations, pooling the network outputs includes summing the network outputs.

[0019] In some implementations, pooling the network outputs includes averaging the network outputs.

[0020] In some implementations, pooling the network outputs includes selecting the largest network output.

[0021] In some implementations, in the sequence of time steps, the time steps are indexed from 1 for the first time step in the sequence to T for the last time step in the sequence, and each substate is assigned an index ranging from 1 to r, and selecting a substate from the r substates includes selecting a substate having an index equal to the index of the time step modulo r.

[0022] In some implementations, the LSTM neural network comprises multiple LSTM layers.

[0023] In some implementations, using an LSTM neural network to process a current value of the selected substate and a current value of the network input for the time step to update the current value of the selected substate and to generate a network output for the time step according to the current values ​​of the set of LSTM network parameters includes setting an internal state of the LSTM neural network to the current value of the selected substate for processing the network input at the time step.

[0024] The above aspects may be implemented in any convenient form. For example, a suitable computer program capable of implementing the aspects and implementations may be carried on a suitable carrier medium, which may be a tangible carrier medium (e.g., a disk) or an intangible carrier medium (e.g., a communication signal). The aspects may be implemented using a suitable apparatus, which may take the form of a programmable computer that executes the computer program.

[0025] Particular embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages: The structure of the action selection system as described herein, including one or both of a separated manager-worker system structure and an extended LSTM network, allows the system to be effectively trained to perform long time scale credit assignment (i.e., to recognize the influence of previous states and actions over many previous time steps on the reward received). In particular, the structure of the action selection system allows gradients to be back-propagated (by a reinforcement learning algorithm) through the system in response to rewards received at a given time step, so as to take into account states and actions over many previous time steps in adjusting the system's parameters. Because the structure of the action selection system facilitates long time scale credit assignment, training a action selection system as described herein may consume less computational resources (e.g., memory and computing power) than training other action selection systems having different structures. In particular, an action selection system such as that described herein may require fewer training iterations to perform a task with a particular performance (e.g., the long-term, time-discounted reward that an agent receives while performing the task) than another system having a different structure.

[0026] Moreover, the manager system of the behavior selection system as described herein generates directional goals for the worker system rather than absolute goals. Specifically, the worker system is trained to generate actions that move the manager system's latent state representation in favorable directions in the latent state space rather than being trained to explicitly move it closer to an absolute goal in the latent state space. The worker system can more effectively generate actions that move the manager system's latent state representation in a particular direction in the latent state space than actions that move it closer to an absolute goal in the latent state space. Thus, by using directional goals rather than absolute goals, the behavior selection system as described herein can achieve better performance (e.g., agents receive larger long-term time-discounted rewards) and faster training (thereby consuming fewer computational resources) compared to systems that use absolute goals.

[0027] An extended long short-term memory (LSTM) neural network as described herein can effectively preserve its internal memory through many time steps, while still being able to generate outputs at each time step and learn from all inputs. Thus, an extended LSTM neural network can generate outputs for sequences where a given output may depend on inputs much earlier in the input sequence. Thus, incorporating an extended LSTM network into an action selection system as described herein can improve the performance of an agent interacting with an environment based on an action selected by the action selection system (e.g., by improving the agent's performance in a particular task as reflected in an increase in the long-term time-discounted reward the agent receives while performing the particular task).

[0028] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0029] [Figure 1] 1 is a diagram of an example behavior selection system. [Diagram 2] 4 is a flow diagram of an example process for selecting an action from a predefined set of actions using the action selection system. [Diagram 3] 13 is a flow diagram of an example process for generating a final goal vector for a time step using a manager system. [Figure 4] 1 is a flow diagram of an example process for generating a behavior score for a time step using a worker system. [Diagram 5] 1 is a flow diagram of an example process for training a behavior selection system. [Figure 6] 1 is a flow diagram of an example process for generating a network output using an extended long short-term memory (LSTM) network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Like reference numbers and designations in the various drawings indicate like elements.

[0031] 1 illustrates an example behavior selection system 100. The behavior selection system 100 is an example of a system implemented as a computer program in one or more computers at one or more locations that implements the systems, components, and techniques described below.

[0032] The behavior selection system 100 selects, at each of a number of time steps, a behavior 144 to be performed by an agent 142 interacting with an environment 140. As the agent 142 interacts with the environment 140, the system 100 receives observations 136 that characterize a current state of the environment 140, and selects a behavior 144 to be performed by the agent 142 in response to the received observations 136.

[0033] In some implementations, the environment 140 is a simulated environment and the agent 142 is implemented as one or more computer programs that interact with the simulated environment by navigating through the simulated environment. For example, the simulated environment can be a video game and the agent 142 can be a simulated user that plays the video game by navigating through the simulated environment. As another example, the simulated environment can be a motion simulation environment, such as a driving or flight simulation, and the agent 142 is a simulated vehicle that navigates through the motion simulation environment. The behaviors 144 in these cases can be control inputs for controlling the simulated user or the simulated vehicle.

[0034] In some other implementations, the environment 140 is a real-world environment and the agent 142 is a mechanical agent that interacts with the real-world environment. For example, the agent 142 may be a robot that interacts with the environment 140 to accomplish a particular task. As another example, the agent 142 may be an autonomous or semi-autonomous vehicle that navigates through the environment. The behaviors 144 in these cases may be control inputs for controlling the robot or autonomous vehicle.

[0035] In some of these implementations, the observations 136 may be generated by or derived from a sensor of the agent 142. For example, the observations 136 may be RGB images captured by a camera of the agent 142. As another example, the observations 136 may be derived from data captured by a laser sensor of the agent 142. As another example, the observations 136 may be hyperspectral images captured by a hyperspectral sensor of the agent 142.

[0036] At each time step, the system 100 receives an extrinsic reward 138 that is based on the current state of the environment 140 at the time step and the actions 144 of the agent 142. Generally, the extrinsic reward is a numerical value that reflects the effectiveness of the process of performing the actions 144 of the agent 142 in accomplishing a particular task. For example, the system 100 may receive, for a given time step, an extrinsic reward 138 that is based on the progress of the agent 142 in accomplishing one or more goals. For example, the agent's goal may be to navigate to a desired location in the environment 140.

[0037] The system 100 includes an encoding neural network 120 configured to receive observations 136 for a time step and process the observations according to current values ​​of a set of encoding neural network parameters to generate as output an intermediate representation 118 of the observations 136 for the time step. The intermediate representation 118 may be an ordered set of numerical values ​​(e.g., a vector or a matrix) that represents the observations 136 in an intermediate state space (e.g., a Euclidean space to which the intermediate representation 118 belongs) for the time step. In particular, when the observations 136 are images, the encoding neural network 120 may be a convolutional neural network.

[0038] The system 100 includes a manager system 102 and a worker system 116. The manager system 102 is configured to generate an abstract goal (i.e., an initial goal vector 110) at each time step. The worker system 116 is configured to generate a behavior score 132 that causes an agent 142 to perform an action to achieve the abstract goal generated by the manager system 102 at each time step.

[0039] The manager system 102, which is provided with the intermediate representation 118 for the time step as input from the system 100, is configured to receive the intermediate representation 118 for the time step as input, process the intermediate representation according to the current values ​​of a set of manager system parameters, and generate a final target vector 114 for the time step as output.

[0040] The manager system includes a latent representation neural network 104 configured to receive as input the intermediate representation 118 for the time step and process the intermediate representation according to current values ​​of a set of latent representation neural network parameters to generate as output a latent representation 106 for the time step. The latent representation 106 is an ordered set of numerical values ​​(e.g., vector or matrix) that represents the intermediate representation 118 in a latent state space (e.g., a Euclidean space to which the latent representation 106 belongs) for the time step and (indirectly) represents an observation 136. In some implementations, the latent representation neural network 104 is a feedforward neural network (e.g., a multi-layer perceptron). Thus, the manager system then uses the latent state space representation to operate (f) using, for example, the latent representation neural network, so that it can define a goal. Mspace) may map the intermediate representation to a latent representation of the current state in a latent state space. Goals may be represented by goal vectors in what may by definition be termed a goal space. In some implementations, the latent state space may be learned, as described later. Using such latent representations may facilitate setting higher level goals, and thus facilitating learning of tasks that are complex, include sub-goals, and / or have delayed rewards.

[0041] The manager system 102 includes a goal regression neural network 108 configured to process the latent representation 106 for the time step and the internal state of the goal regression neural network 108 for the time step. The goal regression neural network 108 updates the internal state of the goal regression neural network 108 to generate an initial goal vector 110 for the time step as an output. The initial goal vector 110 for the time step is an ordered set of numbers logically arranged into a vector. The goal regression neural network 108 may be implemented as any suitable recurrent neural network model. In some implementations, the goal regression neural network 108 is an LSTM network. In some implementations, the goal regression neural network 108 is an extended LSTM network. The extended LSTM network is an LSTM network that generates a network output but processes and updates only a subset of its internal state for each time step. An example process for generating a network output (e.g., an initial goal vector 110) using an extended LSTM network is described with reference to FIG. 6.

[0042] The manager system 102 includes a goal pool engine 112 that pools and combines the initial goal vector for a time step 110 and the initial goal vector for one or more previous time steps to generate a final goal vector for a time step 114. The final goal vector for a time step 114 is an ordered collection of numerical values ​​logically arranged into a vector. The final goal vector 114 changes smoothly between time steps due to the pooling operations performed by the goal pool engine 112. An exemplary process for generating a final goal vector for a time step using the manager system is described with reference to FIG. 3.

[0043] The system 100 provides the intermediate representation for the time step 118 and the final goal vector for the time step 114 (generated by the manager system 102) as input to the worker system 116.

[0044] The worker system 116 is configured to process the input according to the current values ​​of a set of worker system parameters to produce an output behavior score 132. The system uses the behavior score 132 to select an action 144 to be performed by an agent 142 for a time step.

[0045] The behavior score 132 includes an individual score for each behavior in the predefined behavior set (i.e., behaviors that may be adopted by the agent 142). In some implementations, the system 100 selects the behavior 144 to be performed by the agent 142 as the behavior with the highest corresponding score according to the behavior score 132. In some other implementations, the system determines the probability of each behavior in the predefined behavior set by processing the behavior score 132 through a softmax function. In these implementations, the system may select the behavior 144 to be performed by the agent by sampling from a probability distribution over the predefined behavior set determined based on the behavior score 132.

[0046] The worker system 116 includes a behavior score regression neural network 122 configured to process the intermediate representation 118 for the time step and the internal state of the behavior score regression neural network 122 for the time step. The behavior score regression neural network 122 updates the internal state of the behavior score regression neural network 122 to generate a set of behavior embedding vectors 124 for the time step as output. The set of behavior embedding vectors 124 includes an embedding vector for each behavior of a given set of behaviors. A behavior embedding vector is, by definition, an ordered set of numbers arranged into a vector in an embedding space. Thus, in a behavior score regression neural network implementation of the worker system, the intermediate representation may be mapped to the embedding space by the worker system. The behavior score regression neural network 122 may be implemented as any suitable recurrent neural network model. In some implementations, the behavior score regression neural network 122 is implemented as a LSTM network. In some implementations, the behavior score regression neural network 122 is implemented as an extended LSTM network.

[0047] The worker system 116 includes a projection engine 126 configured to receive the final target vector for the time step 114 generated by the manager system 102, process the final target vector for the time step 114 according to the current values ​​of a set of projection engine parameters, and generate a target embedding vector 130 as an output in the embedding space. The target embedding vector 130 is an ordered collection of numbers arranged into a vector. In general, the target embedding vector 130 has a smaller dimensionality than the final target vector 114. In some implementations, the dimensionality of the target embedding vector 130 may be 10 times or more (i.e., an order of magnitude or more) smaller than the dimensionality of the final target vector 114.

[0048] The worker system 116 includes an adjustment engine 128 configured to receive as input the behavior embedding vector for the time step 124 and the target embedding vector for the time step 130, and generate as output a behavior score 132. Thus, the behavior embedding vector is adjusted using the target embedding vector, for example, to weight the behavior embedding vector, i.e., is adjusted depending on the value of the target embedding vector. An exemplary process for generating a behavior score for a time step using a worker system is described with reference to FIG.

[0049] The system selects actions 144 to be performed by the agents 142 based on the action scores 132 generated by the worker system 116. An exemplary process for selecting an action from a predefined set of actions using the action selection system is described with reference to FIG.

[0050] The system 100 includes a training engine 134 that is configured to train the system 100 by adjusting parameters of the system 100, including parameters of the manager system 102, the worker systems 116, and the encoding neural network 120, according to a reinforcement learning training methodology.

[0051] In general, the worker system 116 is trained by the training engine 134 to generate behavior scores 132 that maximize a long-term time-discounted combination of an extrinsic reward 138 received as a result of the agent 142 performing a selected behavior 144 and an intrinsic reward determined by the training engine 134. The training engine 134 determines the intrinsic reward based on how well the behavior scores 132 generated by the worker system 116 move the latent representation 106 in a direction in the latent state space given by the initial goal vector 110 generated by the manager system 102. The manager system 102 is trained by the training engine 134 to generate initial goal vectors 110 that result in behavior scores 132 that encourage the selection of actions 144 that move the latent representation 106 in favorable directions in the latent state space (e.g., directions that will receive a larger long-term time-discounted extrinsic reward 138). An exemplary process for training a behavior selection system is described with reference to FIG. 5.

[0052] 2 is a flow diagram of an exemplary process for selecting an action from a predefined set of actions using a behavior selection system. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, a behavior selection system (e.g., behavior selection system 100 of FIG. 1) suitably programmed in accordance with this specification may perform process 200.

[0053] The system receives observations characterizing a current state of the environment (202). In some implementations, the environment is a simulated environment and the agent is implemented as one or more computer programs that interact with the simulated environment by navigating through the simulated environment. In some other implementations, the environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment. In some implementations, the observations may be generated by or derived from sensors of the agent.

[0054] The system generates (204) intermediate representations of the observations for the time steps. Specifically, the system is configured to provide the observations as inputs to an encoding neural network, which processes the observations according to current values ​​of a set of encoding neural network parameters, and generate intermediate representations of the observations for the time steps as output. The intermediate representations are ordered sets of numerical values ​​(e.g., vectors or matrices) that represent the observations for the time steps in an intermediate state space. In particular, when the observations are images, the encoding neural network can be a convolutional neural network. Values ​​of the set of encoding neural network parameters are determined during training of the system by a training engine.

[0055] The system provides the intermediate representation for the time step to a manager system, which processes the intermediate representation for the time step to generate a final goal vector for the time step as output (206). An example process for generating a final goal vector for a time step using the manager system is described below with reference to FIG.

[0056] The system provides the intermediate representation for the time step and the final goal vector for the time step (generated by the manager system) as input to the worker system. The worker system processes the input for the time step and generates an action score for the time step as output (208). An exemplary process for generating an action score for a time step using a worker system is described with reference to FIG.

[0057] The system selects 210 an action from the predefined action set based on the action scores generated by the worker system. The action scores include an individual score for each action in the predefined action set. In some implementations, the system selects the action to be performed by the agent as the action with the highest corresponding score according to the action scores. In some other implementations, the system determines the probability of each action in the predefined action set by processing the action scores through a softmax function. In these implementations, the system may select the action to be performed by the agent by sampling from a probability distribution over the predefined action set determined based on the action scores.

[0058] 3 is a flow diagram of an example process for generating a final target vector for a time step using a manager system. For convenience, the process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a manager system (e.g., manager system 102 of FIG. 1) suitably programmed in accordance with this specification can perform the process 300.

[0059] The system receives an intermediate representation of the current state of the environment with respect to a time step 302. Typically, the intermediate representation of the current state of the environment is generated by an encoding neural network.

[0060] The system generates a latent representation for the time step by providing the intermediate representation for the time step as an input to a latent representation neural network (304). The latent representation neural network is configured to process the intermediate representation for the time step according to current values ​​of a set of latent representation neural network parameters to generate a latent representation for the time step as an output. In some implementations, the latent representation neural network is a feedforward neural network (e.g., a multi-layer perceptron). The latent representation for the time step is an ordered set of numerical values ​​(e.g., a vector or matrix) that represent intermediate observations and (indirectly) represent observations in a latent state space for the time step. Values ​​for the set of latent neural network parameters are determined during training of the system by a training engine.

[0061] The system generates an initial target vector for the time step (306). Specifically, the system provides the latent representation for the time step generated by the latent representation neural network to a goal regression neural network. The goal regression neural network is configured to process the latent representation for the time step and an internal state of the goal regression neural network for the time step. The goal regression neural network updates the internal state of the goal regression neural network to generate the initial target vector for the time step as an output, i.e.

number

number

number

[0062] The system generates a final target vector for the time step by pooling the initial target vector for the time step generated by the target regression neural network and the initial target vector for one or more previous time steps generated by the target regression neural network (308). In some implementations, the system pools by summing the initial target vector generated for the current time step and the initial target vector for one or more previous time steps. However, the target vectors may be combined in many other ways. The final target vector for the time step is an ordered collection of numerical values ​​arranged into a vector. The final target vector changes smoothly between time steps due to the pooling operation.

[0063] 4 is a flow diagram of an example process for generating behavioral scores for a time step using a worker system. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a worker system (e.g., worker system 116 of FIG. 1) suitably programmed in accordance with this specification may perform process 400.

[0064] The system receives an intermediate representation of the current state of the environment for a time step and a final goal vector 402. Typically, the intermediate representation of the current state of the environment is generated by an encoding neural network, and the final goal vector for a time step is generated by a manager system.

[0065] The system generates a behavior embedding vector for the time step (404). Specifically, the system provides the intermediate representation for the time step generated by the encoding neural network to a behavior score regression neural network. The behavior score regression neural network is configured to process the intermediate representation for the time step and an internal state of the behavior score regression neural network for the time step. The behavior score regression neural network updates the internal state of the behavior score regression neural network to generate a behavior embedding vector for the time step as an output, i.e.

number

number

number

[0066] The system generates a target embedding vector for the time step based on the final target vector for the time step (406). Specifically, the system provides the final target vector for the time step to a projection engine, which is configured to receive the final target vector for the time step and process the final target vector for the time step according to the current values ​​of the set of projection engine parameters to generate a target embedding vector for the time step as an output. In general, the target embedding vector has a smaller dimensionality than the final target vector. In some implementations, the dimensionality of the target embedding vector may be 10 times or more (i.e., an order of magnitude or more) smaller than the dimensionality of the final target vector. In some implementations, the operation of the projection engine may be given by the following equation: w t =φG t

number

[0067] The system generates a behavior score for the time step (408). Specifically, the system provides the goal embedding vector for the time step and the behavior embedding vector for the time step as inputs to a tuning engine, and the tuning engine is configured to combine the inputs to generate a behavior score for the time step as output. In some implementations, the tuning engine combines the goal embedding vector for the time step and the behavior embedding vector for the time step by matrix vector multiplication to generate a behavior score for the time step. π t =U t w t where the matrix U t Each row of is an action embedding vector for a time step, and w t is the target embedding vector for the time step, and π t is the vector of action scores for time steps.

[0068] 5 is a flow diagram of an exemplary process for training a behavior selection system. For convenience, process 500 is described as being performed by an engine including one or more computers located at one or more locations. For example, a training engine (e.g., training engine 134 of FIG. 1) suitably programmed in accordance with this specification may perform process 500.

[0069] The engine receives extrinsic rewards for the time step (502). The received extrinsic rewards are based on the current state of the environment and the actions of the agent at the time step. Typically, the extrinsic rewards are numerical values ​​that reflect the effectiveness of the agent's efforts to accomplish a particular task. In some cases, the engine may receive extrinsic rewards for a given time step that are based on the progress of the agent in accomplishing one or more objectives. For example, an objective of the agent may be to navigate to a desired location in the environment.

[0070] The engine determines an intrinsic reward for the time step (504). Typically, the engine determines the intrinsic reward based on how well the behavior scores generated by the worker systems move the latent representations (e.g., generated by the manager system in 304) in a direction in the latent state space given by the initial goal vector generated by the manager system (e.g., in 306). For example, the training engine may determine the intrinsic reward for time step t by:

number

number

[0071] The engine adjusts parameters of the worker system based on the extrinsic and intrinsic rewards (506). Specifically, the engine adjusts parameters of the tuning engine, the projection engine, the behavior score regressive neural network, and the encoding neural network to cause the worker system to generate behavior scores that maximize a long-term time-discounted combination of the extrinsic rewards received as a result of the agent performing the selected behavior and the intrinsic rewards determined by the training engine. For example, the long-term time-discounted combination may be given by the following formula:

number

number

number

number

[0072] The engine updates the parameters of the manager system based on the extrinsic reward (508). Specifically, the engine adjusts the parameters of the goal regression neural network, the parameters of the latent representation neural network, and the parameters of the encoding neural network. By adjusting the parameters of the manager system, the manager system generates an initial goal vector that results in an action score that encourages the selection of an action that moves the latent representation of the manager system in a favorable direction in the latent state space. A direction in the latent state space is considered favorable, for example, if moving the latent representation in that direction will receive a larger long-term time-discounted extrinsic reward. In some implementations, the parameters of the manager system may be updated by gradient ascent, where the gradient at time step t+c is given by the following equation:

number

number

number

[0073] 6 is a flow diagram of an example process for generating a network output using an extended LSTM network. For convenience, the process 600 is described as being performed by a neural network including one or more computers located at one or more locations. For example, a neural network appropriately programmed in accordance with this specification (e.g., the goal regression neural network 108 of FIG. 1) can perform the process 600.

[0074] In general, an extended LSTM network is an LSTM network that includes a sequence of one or more LSTM network layers, each maintaining an internal state that is divided into multiple independent substates. The internal state of an LSTM network layer is an ordered set of numerical values ​​(e.g., represented as a vector or matrix). A substate of an LSTM network layer's internal state is a subset of the internal state's ordered set of numerical values.

[0075] The network receives network inputs for the time steps (602). Typically, the network receives a respective network input for each time step in the sequence of time steps. For example, the network inputs for the time steps may be latent representations for the time steps generated by a latent representation neural network.

[0076] Each LSTM network layer selects a substate for the time step (604). The time steps in the sequence of time steps may be indexed starting with 1 for the first time step in the sequence and up to T for the last time step in the sequence, and for a particular LSTM network layer, each substate of the LSTM network layer may be assigned an index ranging from 1 to r. A particular LSTM network layer may select a substate for the time step having a substate index equal to the index of the time step modulo r. For example, if each time step is assigned an index from 1 to 10, and each substate is assigned an index from 1 to 3, then the substate selected for the time step assigned index 1 will be the substate assigned index 1, and the substate selected for the time step assigned index 5 will be the substate assigned index 2.

[0077] The current value of the substate may include an output value of the node of the LSTM network layer indexed by that substate. For each LSTM network layer, the LSTM network layer processes the current value of the selected substate for the time step and the input for the time step according to a set of LSTM network layer parameters, updates the current value of the selected substate for the time step, and generates an output for the time step (606). The first LSTM network layer in the sequence receives the network input for the time step. Each subsequent LSTM network layer receives as an input for the time step the output of the previous LSTM network layer in the sequence of time steps. Each LSTM network layer may perform the operations of:

number

number

number

number

[0078] The network determines a network output for the time step (608). In some implementations, the network determines the network output for the time step to be the final LSTM network layer output for the time step. In some other implementations, the network determines the network output for the time step by pooling, i.e., combining, the final LSTM network layer output for the time step with up to a predetermined number of outputs of previous time steps. The network may pool the final LSTM network layer outputs by summing, by averaging, by selecting the final LSTM network layer output with the largest norm, or by any other suitable pooling method. This makes it easier for the extended LSTM network to maintain memory over a longer period of time.

[0079] This specification uses the term "configured" in the context of systems and computer program components. With respect to a system of one or more computers, being configured to perform a particular operation or act means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform the operation or act. With respect to a program of one or more computers, being configured to perform a particular operation or act means that instructions contained in the program or programs, when executed by a data processing device, cause the device to perform the operation or act.

[0080] The subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. The subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a tangible, non-volatile storage medium for execution by or to control the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random access memory device or a serial access memory device, or one or more combinations thereof. Alternatively, or in addition, the program instructions may be encoded on a human-generated and propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal to encode information for transmission to an appropriate receiving device for execution by the data processing device.

[0081] The term "data processing apparatus" refers to data processing hardware and encompasses any kind of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or further include special purpose logic circuitry, such as an FPGA (field programmable gate array), or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or any combination of one or more of these.

[0082] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script, or code, may be written in any type of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be arranged in any form, whether included as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may correspond to a file in a file system, but this is not required. A program may be stored in a single file dedicated to the program, part of a file holding other programs or data, e.g. one or more scripts stored in a markup language document, or in multiple integrated files, e.g. files storing one or more modules, subprograms, or code portions. A computer program may be arranged to be executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0083] As used herein, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine, and in other cases, multiple engines may be installed and operating on the same computer.

[0084] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs that perform functions by operating on input data to generate output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0085] A computer suitable for executing a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. In general, the central processing unit will receive instructions and data from a read-only or random-access memory, or both. The essential elements of a computer are a central processing unit for executing or carrying out instructions and one or more storage devices for storing instructions and data. The central processing unit and storage devices may be supplemented by, or incorporated in, special-purpose logic circuitry. In general, a computer will also include one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or be operatively coupled to receive data from or transfer data to, or both. However, a computer need not have such devices. Moreover, a computer may be incorporated in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a mobile storage device (e.g., a Universal Serial Bus (USB) flash drive), to name just a few.

[0086] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile storage devices, media, and storage devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0087] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and a pointing device, e.g., a mouse or trackball, through which the user can provide input to the computer. Other types of devices may be used to provide interaction with the user as well, e.g., feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. In addition, the computer may interact with the user by sending and receiving documents to and from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a response message in return from the user.

[0088] A data processing device for implementing machine learning models may also include dedicated hardware accelerator units, for example for handling the common numerical computational parts of machine learning training or production, i.e. inference, workload.

[0089] The machine learning model may be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0090] An embodiment of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., as a data server, or a middleware component, e.g., an application server, or a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an application that allows a user to interact with an embodiment of the subject matter described herein, or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0091] A computing system may include clients and servers. Generally, clients and servers are remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to a user device acting as a client to display the data and receives user input from a user interacting with the device. Data generated at the user device, e.g., a result of the user interaction, may be received at the server from the device.

[0092] Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or limitations on what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Certain features described in the context of individual embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in the context of one embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although each feature has been described above as working in a combination and may even be claimed as such from the outset, one or more features of a claimed combination may in some cases be excluded from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0093] Similarly, although operations are depicted in a particular order in the figures and listed in a particular order in the claims, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In certain environments, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the foregoing embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be incorporated together in one software product or packaged into multiple software products.

[0094] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0095] 100 Action Selection System 102 Manager System 104 Latent Representation Neural Network 106 Latent expression 108 Goal Regression Neural Network 110 Initial target vector 112 Target Pool Engine 114 Final Target Vector 116 Worker System 118 Intermediate Representation 120 Coding Neural Networks 122 Behavioral Score Regression Neural Network 124 Behavioral Embedding Vectors 126 Projection Engine 128 Adjustment Engine 130 Target Embedding Vector 132 Behavioral Score 134 Training Engine 136 Observations 138 Extrinsic rewards 140 Environment 142 Agent 144 Action

Claims

1. 1. A system for selecting an action to be performed by an agent that interacts with an environment by performing actions, comprising: The system includes one or more computers and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing a plurality of instructions that, when executed by the one or more computers, cause the one or more computers to implement a worker neural network system and a behavior selection subsystem; The worker neural network system comprises: At each of a plurality of time steps, receiving a goal representation defining an objective to be achieved as a result of actions performed by the agent within the environment, the goal representation being based at least in part on a state of the environment with respect to the time step and / or one or more previous time steps; generating a respective behavior score for each behavior in the behavior set based at least in part on the target representation and the state of the environment for the time step; It is configured to The action selection subsystem includes: At each of the plurality of time steps, using the action score to select an action from the set of actions for the agent to perform at the time step. A system configured to:

2. The method further includes a coding neural network system having a set of coding neural network parameters, the coding neural network system comprising: receiving observations characterizing a current state of the environment; processing the observations according to current values ​​of a set of encoding neural network parameters to generate an intermediate representation of the observations; outputting the intermediate representation; The system of claim 1 configured to:

3. 3. The system of claim 2, wherein the encoding neural network system comprises a convolutional neural network.

4. the worker neural network system is configured to receive the intermediate representation; The system of claim 2 , wherein the generation of the respective behavior scores by the worker neural network system is based at least in part on the intermediate representation.

5. the worker neural network system includes a projection engine having a set of projection engine parameters; The projection engine comprises: receiving the target expression; processing the target representation according to current values ​​of the set of projection engine parameters to generate a target embedding vector; outputting the target embedding vector; configured to: The system of claim 4 , wherein the generation of the respective behavior scores for each behavior in the behavior set is based at least in part on the target embedding vector.

6. The worker neural network system comprises: mapping the intermediate representation to a respective action embedding vector for each action in the action set; The system of claim 5 configured to:

7. the worker neural network system includes a coordination engine; The coordination engine, receiving the respective behavior embedding vectors; receiving the target embedding vector; generating a respective behavior score for each behavior in the behavior set based on the respective behavior embedding vector for the behavior and the target embedding vector; The system of claim 6 configured to:

8. generating said respective behavior scores for the behaviors, adjusting the action embedding vector for the action dependent on the goal embedding vector. The system of claim 7, comprising:

9. generating said respective behavior scores for the behaviors, combining said respective action embedding vectors with said goal embedding vectors. The system of claim 7, comprising:

10. a manager neural network system; The manager neural network system comprises: At each of a plurality of time steps, generating a latent representation in a latent space of a current state of the environment at the time step; generating a final goal representation for said time step using at least said latent representation of said current state of said environment at said time step; The system of claim 1 configured to:

11. the manager neural network system is configured to receive from the encoding neural network an intermediate representation of observations characterizing a current state of the environment; The system of claim 10 , wherein the generation of the latent representation is based at least in part on the intermediate representation.

12. 12. The system of claim 11, wherein generating the latent representation comprises processing the intermediate representation using a latent representation neural network.

13. The system of claim 12 , wherein the latent representation neural network comprises a feedforward neural network.

14. generating, at each of the plurality of time steps, the final goal representation for the time step using at least the latent representation of the current state of the environment at the time step; generating an initial goal representation for the time step based at least in part on the latent representation of the current state of the environment at the time step; combining the initial target representation for the time step with initial target representations for one or more previous time steps to generate the final target representation for the time step; The system of claim 10, comprising:

15. generating, at each of the plurality of time steps, the initial target representation for the time step, processing the latent representation for the time step using a target neural network having a set of target neural network parameters; The target neural network is receiving the latent representation; processing the latent representation according to current values ​​of the set of target neural network parameters; The system of claim 14 configured to:

16. The system of claim 1 , wherein selecting the action at each of the plurality of time steps comprises selecting the action having a highest action score.

17. the worker neural network system is trained to generate behavioral scores that maximize a combination of time-discounted rewards; 10. The system of claim 1, wherein each reward is a combination of an extrinsic reward received as a result of the agent performing a selected behavior and an intrinsic reward that depends on a goal representation.

18. 11. The system of claim 10, wherein the manager neural network system is trained to generate goal representations that result in behavior scores that encourage selection of behaviors that increase extrinsic rewards received as a result of the agent performing the selected behavior.

19. 1. A method implemented by one or more computers for selecting a behavior from a set of behaviors to be performed by an agent that interacts with an environment by performing the behaviors, comprising: At each of a plurality of time steps, generating a goal representation that defines an objective to be achieved as a result of actions performed by said agent within said environment, said goal representation being based at least in part on a state of said environment with respect to said time step and / or one or more previous time steps; generating a respective behavior score for each behavior in the behavior set based at least in part on the target representation and the state of the environment for the time step; using the action score to select an action from the set of actions for the agent to perform at the time step; A method comprising:

20. One or more non-transitory computer-readable media having instructions stored thereon, The instructions, when executed by one or more computers, cause the one or more computers to perform a plurality of operations for selecting a behavior from a set of behaviors to be performed by an agent that interacts with an environment by performing a behavior, the plurality of operations comprising, at each of a plurality of time steps: generating a goal representation defining an objective to be achieved as a result of actions performed by the agent within the environment, the goal representation being based at least in part on a state of the environment with respect to the time step and / or one or more previous time steps; generating a respective behavior score for each behavior in the behavior set based at least in part on the target representation and the state of the environment for the time step; using the action score to select an action from the set of actions for the agent to perform at the time step; and A non-transitory computer-recorded medium,

Citation Information

Patent Citations

  • Learning control device, learning control method, and program

    JP2006268812A

  • Methods for controlling and / or regulating a technical system using computer assistance.

    JP2010524104A

  • Selecting reinforcement learning actions using goals and observations

    US20160292568A1