Attentional neural network with short-term memory units

Abstract

To provide a system for controlling an agent interacting with an environment to perform a task.SOLUTION: The system includes an action selection neural network configured to generate action selection outputs that are used to select actions to be performed by an agent. The action selection neural network includes: an encoder subnetwork configured to generate encoded representations of current observations; an attention subnetwork configured to generate attention subnetwork outputs using an attention mechanism; a recurrent subnetwork configured to generate recurrent subnetwork outputs; and an action selection subnetwork configured to generate action selection outputs that are used to select actions to be performed by the agent in response to the current observations.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 146,361, filed on 5 February 2021. The disclosures of the prior application are by reference to and incorporated into the disclosures of this application.

[0002] This specification relates to reinforcement learning. [Background technology]

[0003] In a reinforcement learning system, an agent interacts with the environment by performing an action selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.

[0004] Some reinforcement learning systems select an action that should be performed by an agent in response to receiving a given observation, based on the output of the neural network.

[0005] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output based on an incoming input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer, or the output layer. Each layer of the network generates an output from an incoming input, depending on the current values ​​of the parameters of its respective set. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Vaswani et al., Attention Is All You Need, arXiv:1706.03762 [Non-Patent Document 2] Parisotto et al., Stabilizing transformers for reinforcement learning, arXiv:1910.06764 [Non-Patent Document 3] Song et al., V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control, arXiv:1909.12238 [Non-Patent Document 4] Kapturowski et al., Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018 [Non-Patent Document 5] DeepMind Lab(https: / / arxiv.org / abs / 1612.03801) [Overview of the project] [Means for solving the problem]

[0007] This specification provides a general description of reinforcement learning systems that control agents interacting with their environment.

[0008] The subject matter described herein can be implemented in particular embodiments to achieve one or more of the following advantages:

[0009] The technique described combines the self-attention mechanism of attention-based neural networks with the memory mechanism of recurrent neural networks, such as long-short-term memory (LSTM) neural networks, to create an action-selection neural network used by reinforcement learning systems to select actions to be performed by an agent. This technique can provide the action-selection neural network with temporally structured information to improve the quality of the action-selection output during or after training, i.e., at runtime. In detail, the technique effectively leverages both the self-attention mechanism, which extracts long-range dependencies, and the memory mechanism, which infers short-term dependencies, to integrate information about the agent's past interactions with the environment across numerous different time scales. As a result, the action-selection neural network can infer about events across multiple time scales and adjust future action-selection policies accordingly.

[0010] Furthermore, the techniques described herein optionally include implementations of trainable gating mechanisms to enable action-selection neural networks to more effectively combine information computed by using attention-based neural networks and recurrent neural networks. This effective combination can be particularly advantageous in complex environments, as it allows for greater flexibility in determining which information should be processed when controlling a robotic agent. Here, the term “gating mechanism” means a unit that forms a dataset based on both the inputs to a neural network and the outputs of the neural network. A trainable gating mechanism is a gating mechanism in which the dataset is further based on one or more tunable parameter values. The gating mechanism may be used, for example, to generate a dataset based on both the inputs to an attention subnetwork and the outputs of the attention subnetwork.

[0011] Therefore, the reinforcement learning system described herein can outperform conventional reinforcement learning systems in controlling agents to perform tasks, for example, by receiving more cumulative external rewards. The reinforcement learning system described herein trains action-selection neural networks faster than conventional reinforcement learning systems that do not utilize self-attention mechanisms, memory mechanisms, or neither. Furthermore, in addition to training the neural network to maximize cumulative rewards, the reinforcement learning system described herein can further improve training by training the action-selection neural network on a controlled learning aid task, thereby augmenting the feedback signals received during training of the action-selection neural network to encourage the learning of representations that assist in obstacle avoidance or trajectory planning. Thus, the reinforcement learning system described herein enables more efficient use of computational resources in training.

[0012] Details of one or more embodiments of the subject matter herein are presented in the accompanying drawings and in the subsequent description. Other features, aspects, and advantages of the subject matter will be evident from the description, drawings, and claims. [Brief explanation of the drawing]

[0013] [Figure 1] This is a diagram illustrating an exemplary reinforcement learning system. [Figure 2] This flowchart shows an exemplary process for controlling an agent. [Figure 3] This flowchart illustrates an exemplary process for determining the update of parameter values ​​in an attention-selection neural network. [Figure 4] This diagram illustrates the process of deciding how to update the parameter values ​​of an attention-selection neural network. [Figure 5]A diagram showing a quantitative example of the performance gain that can be achieved by using the control neural network system described in this specification. **DETAILED DESCRIPTION OF THE INVENTION**

[0014] Like reference numerals and names in the various drawings refer to like elements.

[0015] This specification describes a reinforcement learning system that controls an agent that interacts with an environment by processing data (i.e., "observations") that characterize the current state of the environment at each time step of a number of time steps to select an action to be performed by the agent at that time step.

[0016] At each time step, the state of the environment at that time step depends on the state of the environment at the previous time step and the action performed by the agent at the previous time step.

[0017] In some implementations, the environment is a real-world environment and the agent is a machine agent that interacts with the real-world environment, such as a robot moving within the environment or an autonomous or semi-autonomous land, air, or sea vehicle.

[0018] In these implementations, the observations may include, for example, one or more of an image that captures the observation as the agent interacts with the environment, object position data, and sensor data, such as sensor data from an image sensor, a distance sensor, or a position sensor, or from an actuator.

[0019] For example, in the case of a robot, the observation may include one or more data characterizing the robot's current state, such as joint position, joint velocity, joint force, torque or acceleration, such as gravity-compensated torque feedback, or the overall or relative posture of an item being held by the robot.

[0020] In the case of a robot or other mechanical agent or vehicle, observations may similarly include one or more of the following: position, linear velocity or angular velocity, force, torque or acceleration, and the overall or relative orientation of one or more parts of the agent. Observations may be defined in one, two, or three dimensions, and may be absolute and / or relative observations.

[0021] Furthermore, the observation may include, for example, sensed electronic signals such as motor current or temperature signals, and / or image data or video data from, for example, a camera or LIDAR sensor, such as data from the agent's sensors or data from sensors placed separately from the agent in the environment.

[0022] In these implementations, the action may be a control input for controlling a robot, such as torque on the robot's joints or a high-level control command, or a control input for controlling an autonomous or semi-autonomous ground, air, or sea vehicle, such as torque on the vehicle's control surface or other control elements or a high-level control command.

[0023] In other words, an action may include, for example, positional data, velocity data, or force / torque / acceleration data relating to one or more joints of a robot or one or more parts of another mechanical agent. Action data may further, or alternatively, include electronic control data such as motor control data, or more generally, data for controlling one or more electronic devices in the environment whose control affects the observed state of the environment. For example, in the case of an autonomous or semi-autonomous ground, air, or sea vehicle, an action may include actions that control the vehicle's navigation, e.g., steering, and movement, e.g., braking and / or acceleration.

[0024] In some other application forms, the agent may control actions in a real-world environment, including equipment items, for example, in a data center, a power / water distribution system, or a manufacturing plant or service facility. In this case, observations may relate to the operation of the plant or facility. For example, observations may include observing power consumption or water consumption by equipment, or observing power generation control or power distribution control, or observing resource usage or waste calculation. Actions may include actions that control equipment items in the plant / facility or impose operating conditions on equipment items in the plant / facility, and / or actions that result in changes to settings in the operation of the plant / facility, such as adjusting or turning on / off components of the plant / facility.

[0025] In the case of an electronic agent, observations may include data from one or more sensor-monitoring parts of a plant or service facility, such as current sensors, voltage sensors, power sensors, temperature sensors, and other sensors, and / or electronic signals representing the function of electronic and / or mechanical items of the facility. For example, the real-world environment may be a manufacturing plant or service facility, the observations may relate to the operation of the plant or facility, such as resource usage, such as power consumption, and the agent may control actions or operations in the plant / facility, for example, to reduce resource usage. In some other implementations, the real-world environment may be a renewable energy plant, the observations may relate to the operation of the plant, for example, to maximize current or future planned power generation, and the agent may control actions or operations in the plant to achieve this.

[0026] In another embodiment, the environment may be a chemosynthesis environment or a protein folding environment, such that each state is the respective state of a protein chain or one or more intermediates or precursor chemicals, and the agent is a computer system for determining how to fold the protein chain or how to synthesize the chemicals. In this embodiment, the action is a possible folding action for folding the protein chain or an action for assembling a precursor chemical / intermediate, and the results to be achieved may include, for example, folding the protein so that the protein is stable and the protein performs a particular biological function, or resulting in a reasonable synthetic route for the chemical. In another embodiment, the agent may be a mechanical agent that automatically performs or controls a protein folding action or chemosynthesis step selected by the system without human interaction. Observation may include direct or indirect observation of the state of the protein or chemical / intermediate / precursor and / or may be derived from simulation.

[0027] In some implementations, the environment may be a simulated environment, and the agent may be implemented as one or more computers that interact with the simulated environment.

[0028] The simulated environment may be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent may be a simulated vehicle moving within the motion simulation. In these implementations, the action may be a control input that controls a simulated user or a simulated vehicle.

[0029] In some implementations, the simulated environment may be a simulation of a specific real-world environment. For example, the system may be used to select actions in the simulated environment during the training or evaluation of a control neural network, and after training or evaluation is complete, or both, it may be deployed to control a real-world agent in the real-world environment simulated by the simulated environment. This can avoid unnecessary wear and tear and damage to the real-world environment or the real-world agent, and can allow the control neural network to be trained and evaluated against situations that are rare or difficult to reproduce in the real-world environment.

[0030] In general, in the case of a simulated environment, observations may include simulated versions of the observations described above, or one or more observations of the types described above, and actions may include simulated versions of the actions described above, or one or more actions of the types described above.

[0031] Optionally, in any of the preceding implementations, observations at any given time step may include data from previous time steps that may be useful in characterizing the environment, such as actions performed in the previous time step or rewards received in the previous time step.

[0032] Figure 1 shows an exemplary reinforcement learning system 100. The reinforcement learning system 100 is an embodiment of the system implemented as a computer program on one or more computers in one or more locations where the systems, components, and techniques described later are implemented.

[0033] The system 100 controls agent 102 as it interacts with the environment 104 by causing agent 102 to execute the selected action 106, and by sending control data to agent 102 instructing agent 102 to execute action 106. In some cases, the reinforcement learning system 100 may be implemented on agent 102 or be a component of agent 102, and the control data may be sent to the agent's (multiple) actuators.

[0034] The execution of the selected action 106 by agent 102 generally causes the environment 104 to transition to a series of new states. By repeatedly having agent 102 operate in the environment 104, system 100 can control agent 102 to complete a specified task.

[0035] System 100 includes a control neural network system 110 and one or more memories that store a set of model parameters 118 ("network parameters") of the neural network included in the control neural network system 110.

[0036] At a high level, the control neural network system 110 is configured to process an input containing the current observation 108 characterizing the current state of the environment 104 at each of a number of time steps, according to model parameters 118, in order to generate an action selection output 122 that can be used to select a current action 106 to be performed by agent 102 in response to the current observation 108.

[0037] The control neural network system 110 includes an action selection neural network 120. The action selection neural network 120 is implemented with a neural network architecture that improves the quality of the action selection output 122 by enabling the system to detect events occurring on different time scales and to react to those events. Specifically, the action selection neural network 120 includes an encoder subnetwork 124, an attention subnetwork 128, a gating subnetwork 132 (preferably), a recurrent subnetwork 136, and an action selection subnetwork 140. Each subnetwork can be implemented as a group of one or more neural network layers in the action selection neural network 120.

[0038] In each of the many time steps, the encoder subnetwork 124 receives an encoder subnetwork input containing a current observation 108 that characterizes the current state of the environment 104, and an encoded representation ("Y") of the current observation 108. tThe encoder subnetwork is configured to process the encoder subnetwork input according to the trained parameter values ​​of the encoder subnetwork in order to generate the encoded representation 126. The encoded representation 126 can be in the form of an ordered collection of numbers, for example, a vector or matrix of numbers. For example, the encoded representation 126 subsequently given to the attention subnetwork 128 as input can be an input vector having the respective input values ​​at each of the numerous input positions in the input order. In some implementations, the encoded representation 126 has the same dimensions as the observation 108, and in some other implementations, the encoded representation 126 has fewer dimensions than the observation 108 for reasons of computational efficiency. In some implementations, in addition to giving the encoded representation 126 to the attention subnetwork 128 as input, the system also stores the encoded representation 126 generated at a given time step in a memory buffer or lookup table so that the encoded representation 126 can be used later, for example, in future time steps following that given time step.

[0039] If the observation is an image, the encoder subnetwork 124 can be a convolutional subnetwork configured to process the observation over time steps, for example, a convolutional neural network with residual blocks. In some cases, for example, if the observation involves lower-dimensional data, the encoder subnetwork 124 can further, or instead, include one or more fully connected neural network layers.

[0040] The attention subnetwork 128 is a network comprising one or more attention neural network layers. Each attention layer operates on each input sequence containing each input vector at each of one or more locations (e.g., multiple concatenated input vectors). At each of the many time steps, the attention subnetwork 128 receives an attention subnetwork input containing an encoded representation 126 of the current observation 108 and an encoded representation of one or more previous observations, and at least in part applies the attention mechanism to each encoded representation of the current observation and one or more previous observations to produce an attention subnetwork output ("X"). t The attention subnetwork is configured to process the input to the attention subnetwork according to the trained parameter values ​​of the attention subnetwork in order to generate the attention subnetwork output (X t )130 is the output determined by or otherwise derived from the updated (i.e., "attention-focused") representation of the encoded representation generated by using one or more attention neural network layers.

[0041] More specifically, in addition to the encoded representation 126 of the current observation 108, the attention subnetwork input also includes encoded representations of one or more previous observations characterizing one or more previous states of the environment that immediately preceded the current state of the environment 108. Each encoded representation of a previous observation can take the form of an input vector containing the respective input values ​​at each of a number of input positions in the input sequence. Thus, the attention subnetwork input can be a concatenated input vector consisting of a number of individual input vectors, each corresponding to an encoded representation of an observation of a different previous state of the environment 108 leading up to (and including) the current state.

[0042] In general, the attention layer within the attention subnetwork 128 can be configured in any of a variety of configurations. Examples of attention layer configurations and details of other components of the attention subnetwork are described in more detail by reference in Vaswani et al., Attention Is All You Need, arXiv:1706.03762, and Parisotto et al., Stabilizing transformers for reinforcement learning, arXiv:1910.06764, which are incorporated herein in their entirety. For example, the attention mechanism applied by the attention layer within the attention subnetwork 128 can be a self-attention mechanism, such as a multi-head self-attention mechanism.

[0043] Generally, an attention mechanism maps queries and sets of key-value pairs to an output, where the queries, keys, and values ​​are all vectors derived from the input to the attention mechanism based on their respective matrices. The output is calculated as a weighted sum of the values, where the weights assigned to each value are calculated by a goodness-of-fit function of the queries with the corresponding keys, e.g., the dot product or scaled dot product. Generally, an attention mechanism determines the relationship between two sequences, while a self-attention mechanism relates different positions within the same sequence to determine a transformed version of that sequence as its output. The attention layer input may contain vectors relating to each element of the input sequence. These vectors are used by the self-attention mechanism to determine a new representation of the same sequence for the attention layer output, which provides the self-attention mechanism with input and similarly contains vectors relating to each element of the input sequence. The output of the self-attention mechanism may be used as the attention layer output.

[0044] In some implementations, the attention mechanism is, for example, matrix W Q A query transformation defined by, for example, matrix W K Key transformations defined by, and for example, matrix W VEach value transformation defined by is used to determine, for each input vector X of the input sequence, a sequence to which attention for output is directed, for each query vector Q = XW Q 、key vector K = XW K 、and value vector V = XW V configured to be applied to derive. For example, the attention mechanism may be an inner product attention mechanism applied by applying each query vector to each key vector to determine respective weights for each value vector, and then combining the value vectors using the respective weights to determine the attention layer output for each element of the input sequence. The attention layer output may be scaled by a scaling factor, e.g., the square root of the dimensions of the query and key, to implement scaled dot-product attention. Thus, for example, the output of the attention mechanism may be

[0045]

Number

[0046] determined as , where d is the dimension of the key (and value) vectors. In another implementation, the attention mechanism comprises an "additive attention" mechanism that calculates a fitness function using a feed-forward network with a hidden layer.

[0047] The attention mechanism may implement multi-head attention, i.e., the attention mechanism may apply a number of different attention mechanisms in parallel. Next, these outputs may be combined, if necessary, with a learned linear transformation applied to reduce to the original dimension, e.g., concatenated.

[0048] An attention neural network layer or self-attention neural network layer is a neural network layer that includes an attention mechanism or self-attention mechanism (which operates on the attention layer input to generate an attention layer output). The attention subnetwork 128 may have a single attention layer, or alternatively, a sequence of attention layers in which each attention layer other than the first attention layer receives the output from the preceding attention layer in the sequence as input.

[0049] Furthermore, in this embodiment, the self-attention mechanism can be masked such that any given position in the input sequence does not attract attention to any position in the input sequence that follows that given position. For example, with respect to a subsequent position in the input sequence that follows that given position, the attention weight for that subsequent position is set to 0.

[0050] The recurrent subnetwork 136 is a network comprising one or more recurrent neural network layers, such as one or more long short-term memory (LSTM) layers, one or more gated recurrent unit (GRU) layers, and one or more vanilla recurrent neural network (RNN) layers. The recurrent subnetwork 136 is configured to receive a recurrent subnetwork input at each of a number of time steps and process it according to the trained parameter values ​​of the recurrent subnetwork in order to update the current hidden state of the recurrent subnetwork corresponding to that time step and generate a recurrent subnetwork output.

[0051] In detail, the recurrent subnetwork input is derived from the attention subnetwork output 130. In some implementations, the action selection neural network 120 can directly input the attention subnetwork output 130 to the recurrent subnetwork 136. Alternatively, in some implementations, the action selection neural network 120 can use the gating subnetwork 132 to combine the output 126 of the encoder subnetwork 124 and the output 130 of the attention subnetwork 128. Therefore, the recurrent subnetwork input is derived from the output of the gating subnetwork 132 of the action selection neural network 120 ("Z"). t It is possible that it is 134.

[0052] In some of these implementations, the gating mechanism implemented by the gating subnetwork 132 is a fixed, additive (or concatenating) mechanism, and the gating subnetwork may include an additive (or concatenating) layer configured to receive an encoded representation 126 of the current observation and the attention subnetwork output 130, and to generate an output based on the representation 126 and the output 130. For example, the gating subnetwork may compute the addition (or concatenation) of i) the encoded representation 126 of the current observation and ii) the attention subnetwork output 130 along a predetermined dimension of the received layer input.

[0053] In some of these implementations, the gating mechanism implemented by the gating subnetwork 132 is a learned gating mechanism that facilitates a more effective combination of information that is contained in or otherwise deriveable from the outputs 126 and 130 of the encoder subnetwork 124 and the output 130 of the attention subnetwork 128. In these implementations, the gating subnetwork 132 may include a gated recurrent unit (GRU) layer configured to apply the learned GRU gating mechanism to i) the encoded representation 126 of the current observation and ii) the attention subnetwork output 130, depending on the trained parameter values ​​of the GRU layer, in order to generate a gated output, i.e., a gating subnetwork output 134, which is then given as input to the recurrent subnetwork 136. The combination of the encoded representation 126 and the attention subnetwork output 130 using the GRU gating mechanism will be further described later with reference to Figure 2. In some implementations, a skip ("residual") connection may be configured between the encoder subnetwork 124 and the gating subnetwork 132, and the gating subnetwork 132 may be configured to receive the encoded representation 126 via the skip connection, in addition to directly receiving the attention subnetwork output 130 from the attention subnetwork 128.

[0054] The action selection subnetwork 140 is configured to receive the action selection subnetwork input at each of the many time steps and process the action selection subnetwork input according to the trained parameter values ​​of the action selection subnetwork 140 in order to generate the action selection output 122. The action selection subnetwork input includes the recurrent subnetwork output and, in some implementations, further includes an encoded representation 126 generated by the encoder subnetwork 124. In implementations where the action selection subnetwork input also includes the encoded representation 126, a skip connection may be configured between the encoder subnetwork 124 and the action selection subnetwork 140, and the action selection subnetwork 140 may be configured to receive the encoded representation 126 via the skip connection in addition to directly receiving the recurrent subnetwork output from the recurrent subnetwork 136.

[0055] Next, system 100 uses the action selection output to select an action to be performed by the agent in the current time step. Several examples of using the action selection output to select an action to be performed by the agent are described below.

[0056] In one embodiment, the action selection output 122 may include a numerical probability value for each action in the set of possible actions that can be performed by the agent. The system can select the action to be performed by the agent, for example, by sampling actions according to their probability values, or by selecting the action with the highest probability value.

[0057] In another embodiment, the action selection output 122 may directly define the action to be performed by the robot agent, for example, by defining the torque value to be applied to the joints of the robot agent.

[0058] In another embodiment, the action selection output 122 may include the respective Q-values ​​for each action in the set of possible actions that can be performed by the agent. The system can process the Q-values ​​(for example, using a softmax function) to generate the respective probability values ​​for each possible action, which can be used to select the action that should be performed by the agent (as described above). The system can also select the action with the highest Q-value as the action that should be performed by the agent.

[0059] The Q-value for an action is an estimate of the "return" resulting from the agent performing an action in response to the current observation, and then selecting a future action to be performed by the agent based on the current values ​​of the policy neural network parameters.

[0060] Return refers to a cumulative measure of the "rewards" received by an agent, such as the time-discounted sum of rewards. An agent can receive a reward at each time step, where the reward is defined by a scalar value that characterizes, for example, the agent's progress toward completing an assigned task.

[0061] In some cases, the system can select the actions to be performed by the agent according to the search policy. For example, the search policy may be an ∈-greedy search policy in which the system selects the actions to be performed by the agent according to the action selection output 122 with a probability of 1-∈, and randomly selects actions with a probability of ∈. In this embodiment, ∈ is a scalar value in the range of 0 to 1.

[0062] The control of the agent using the control neural network system 110 will be explained in more detail later with reference to Figure 2.

[0063] In order to more effectively control the agent 102 interacting with the environment 104, the reinforcement learning system 100 can use the training engine 150 to train the action selection neural network 120 to determine the trained values ​​of the parameters 118 of the action selection neural network 120.

[0064] The training engine 150 is configured to train the action selection neural network 120 by repeatedly updating the model parameters 118, namely the parameter values ​​of the encoder subnetwork 124, attention subnetwork 128, gating subnetwork 132, recurrent subnetwork 136, and action selection subnetwork 140, based on the interaction of agent 102 (or another agent) with environment 108 (or another instance of the environment).

[0065] More specifically, the training engine 150 trains the action selection neural network 120 through reinforcement learning and, optionally, through contrastive representation learning. Contrastive representation learning means training the neural network components of the action selection neural network, in particular the attention subnetwork 128, to produce outputs such that when the action selection neural network receives successive pairs of similar inputs (e.g., measured by a similarity measure, such as a distance measure like Euclidean distance), it produces outputs that are more similar to each other than the outputs that the action selection network would produce from more distant inputs compared to pairs of similar inputs. In the embodiments given later, contrastive representation learning is based on training the action subnetwork 128 to receive an input that is a masked form of data produced by the encoder subnetwork 124 at a given time step, and / or produce an output that reconstructs the same data as that produced by the encoder subnetwork 124 at another time step.

[0066] In reinforcement learning, the action-selection neural network 120 is trained based on the agent's interaction with the environment to optimize an appropriate reinforcement learning objective function. The architecture of the action-selection neural network 120 is agnostic with respect to the selection of a strict RL training algorithm, and therefore, RL training can be either on-policy (e.g., one of the RL algorithms described in more detail in Song et al., V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control, arXiv:1909.12238) or off-policy (e.g., one of the RL algorithms described in more detail in Kapturowski et al., Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018).

[0067] As part of this RL training process, observations 108 received by the reinforcement learning system 100 during the interaction are encoded into encoded representations that generate action selection outputs 122. Therefore, learning to generate encoded representations that provide useful information is a crucial factor for the success of RL training. To this end, the training engine 150 evaluates the time-domain controlled learning objective and uses it as a proxy supervision signal for masked prediction training of the attention subnetwork 128, i.e., to train the attention subnetwork 128 to predict a masked portion of the attention subnetwork input. Such a signal aims to learn self-attention-consistent representations that include appropriate information so that the action selection subnetwork 140 effectively incorporates information previously observed (or extracted) by the attention subnetwork 128.

[0068] In some implementations, by utilizing contrasting representation learning techniques to assist in determining parameter value updates, the training engine 150 improves the efficiency of the training process in terms of the amount of computational resources or real time consumed by the training process required to train the action selection neural network 120 to achieve or surpass the performance of the latest technologies in controlling agents to perform a given task.

[0069] The training of the action selection neural network 120 will be explained in more detail later, with reference to Figures 3 and 4.

[0070] Figure 2 is a flowchart of an exemplary process 200 for controlling an agent. For convenience, process 200 is described as being executed by a system of one or more computers located in one or more locations. For example, a well-programmed reinforcement learning system, e.g., the reinforcement learning system 100 in Figure 1, can execute process 200.

[0071] In general, the system can repeatedly execute process 200 to select each action (hereinafter referred to as the "current action") that should be performed by the agent in each state of the environment corresponding to that time step (hereinafter referred to as the "current" time step) (hereinafter referred to as the "current" state).

[0072] The system receives a current observation characterizing the current state of the environment at the current time step and generates an encoded representation of the current observation by using an encoder subnetwork (step 202). For example, the current observation can include an image, a video frame, an audio data segment, a sentence in natural language, or something similar. In some of these embodiments, the observation can also include information derived from a previous time step, such as a previous action performed, a reward received in the previous time step, or both. The encoded representation of the observation can be represented as an ordered collection of numbers, such as a vector or matrix of numbers.

[0073] The system processes an attention subnet input, which includes an encoded representation of the current observation and encoded representations of one or more previous observations, by using the attention subnet to generate an attention subnet output (step 204). The one or more previous observations characterize one or more previous states of the environment that precede the current state of the environment, and thus the encoded representations of the one or more previous observations can be encoded representations generated by using the encoder subnet in one or more time steps preceding the current time step.

[0074] An attention subnetwork may be a neural network comprising one or more attention neural network layers, configured at least in part to generate an attention subnetwork output by applying an attention mechanism, such as a self-attention mechanism, to an encoded representation of the current observation and an encoded representation of one or more previous observations characterizing one or more previous states of the environment. This use of an attention mechanism facilitates, for example, the connection of long-range data dependencies across each observation of a prolonged sequence of different states of the environment.

[0075] More specifically, the attention subnetwork input can include a concatenated input vector consisting of a number of individual input vectors, each corresponding to a different encoded representation of the input value at each of the numerous input positions in the input sequence. To generate the attention subnetwork output, each attention layer included in the attention subnetwork can be configured to receive an attention layer input (which may also be in vector format) for each of one or more layer input positions, and for each particular layer input position in the layer input sequence, to apply an attention mechanism to the attention layer input at that layer input position using one or more queries derived from the attention layer input at that particular layer input position in order to generate the respective attention layer output for that particular layer input position.

[0076] The system generates a combination of i) an encoded representation of the current observation and ii) the attention subnetwork output, and then provides this combination as input to the recurrent subnetwork. To generate this combination, the system can utilize a gating subnetwork configured to apply a gating mechanism to i) the encoded representation of the current observation and ii) the attention subnetwork output, in order to generate the recurrent subnetwork input. For example, the gating subnetwork may include a gated recurrent unit (GRU) layer, which can be configured to apply a GRU gating mechanism of lower complexity compared to an LSTM layer by using fewer layer parameters. In another embodiment, the gating subnetwork may also compute the sum or concatenation of i) the encoded representation of the current observation and ii) the attention subnetwork output.

[0077] Specifically, in the preceding embodiment, the GRU layer is a recurrent neural network layer with fewer parameters than an LSTM, as it provides performance similar to an LSTM with a forget gate, but lacks an output gate. In some implementations, this gating mechanism can be adapted as an update of the GRU layer that is rewound with respect to the depth of the action-selection neural network, rather than being rewound with respect to time. This means that while the GRU layer is a recurrent neural network (RNN) layer, the gating mechanism can instead generate an "updated" combination of inputs received in the gating subnetwork of the action-selection neural network using the same formula that the GRU layer uses to update the hidden state of the GRU layer over time.

[0078] In these implementations, the GRU layer consists of a reset gate r and an update gate z, i.e.,

[0079]

number

[0080] To calculate each of these, a nonlinear function such as sigmoid activation σ() is applied to the weighted combination of the received layer inputs, i.e., the encoded representation Y t and attention subnetwork output X t Applied to and updated hidden state

[0081]

number

[0082] In other words,

[0083]

number

[0084] To generate this, nonlinear functions such as tanh activation tanh() are used in an encoded representation Y t The weighted combination of the reset gate r and the attention subnetwork output X t Apply this to the element-wise product between them, where,

[0085]

number

[0086] This is the weight (or bias) determined from the values ​​of the GRU layer parameters,

[0087]

number

[0088] This represents element-wise multiplication. Next, the GRU layer has a gated output g (l) (x,y) (which can be used as a recurrent subnetwork input) is defined as follows:

[0089]

number

[0090] To generate.

[0091] The system processes the recurrent subnetwork input by using the recurrent subnetwork to generate a recurrent subnetwork output (step 206). The recurrent subnetwork can be configured to receive a recurrent subnetwork input and update its current hidden state by processing the received input, i.e., by processing the current hidden state of the recurrent subnetwork, which was generated by processing the previous recurrent subnetwork input. The updated hidden state of the recurrent subnetwork after processing the recurrent subnetwork input is hereafter referred to as the hidden state corresponding to the current time step. Once the updated hidden state corresponding to the current time step is generated, the system can use the updated hidden state of the recurrent subnetwork to generate a recurrent subnetwork output.

[0092] For example, a recurrent subnetwork can be a recurrent neural network that includes one or more long short-term memory (LSTM) layers. Because LSTM layers are sequential, they can effectively capture short-range dependencies, for example, over a series of recent observations of the environment.

[0093] The system processes the action selection subnetwork input, including the recurrent subnetwork output, by using the action selection subnetwork to generate an action selection output used to select an action to be performed by the agent in response to the current observation (step 208). In some implementations, the action selection subnetwork input also includes an encoded representation. In these implementations, to generate the action selection subnetwork input, the system can compute a concatenation of i) the recurrent subnetwork output and ii) the encoded representation generated by the encoder subnetwork at the current time step.

[0094] Next, the system can cause the agent to perform the selected action, that is, by commanding the agent to perform the action, or by passing control signals to the control system for the agent.

[0095] As mentioned earlier, the system's components can be trained through reinforcement learning combined with contrasting representation learning. In some implementations, the system maintains a replay buffer to aid in training. The replay buffer stores a number of transitions generated as a result of the agent's interaction with the environment. Each transition represents information about the agent's interaction with the environment.

[0096] In these implementations, each transition is an experience tuple that includes: i) a current observation characterizing the current state of the environment at a given point in time; ii) a current action performed by the agent in response to the current observation; iii) a follow-up that characterizes the next state of the environment after the agent has performed the current action, i.e., the state of the environment that has been transitioned to as a result of the agent performing the current action; and iv) a reward received by the agent in response to performing the current action.

[0097] In short, in these implementations, RL training can include iteratively sampling batches of one or more transitions from a replay buffer, and then training an action selection neural network on the sampled transitions by using an appropriate reinforcement learning algorithm. During each RL training iteration, the system can decide to update the current values ​​of the action selection network parameters by processing the current observations contained in each sampled transition using the action selection neural network according to the current parameter values ​​of the action selection neural network in order to generate an action selection output, determining the reinforcement learning loss based on the action selection output, and then calculating the gradient of the reinforcement learning loss with respect to the action selection neural network parameters.

[0098] Figure 4 illustrates the decision to update the parameter values ​​of an attention selection neural network. As shown, the system can determine, at each time step, the respective reinforcement learning loss, for example, the RL loss 410A, with respect to the action selection output generated in time step 402A.

[0099] Contrastive representation learning, which can be used to support RL training in order to improve the efficiency of training data, will be explained further later.

[0100] Figure 3 is a flowchart of an exemplary process 300 for determining the update of parameter values ​​in an attention selection neural network. For convenience, the process 300 is described as being performed by one or more computer systems located in one or more locations. For example, a well-programmed reinforcement learning system, e.g., the reinforcement learning system 100 in Figure 1, is capable of performing the process 300.

[0101] In detail, the system can repeatedly run process 300 to train the encoder and attention subnetwork of the action selection neural network to produce high-quality encoded representations (e.g., informative, predictive, or both) and attention subnetwork outputs, respectively, which facilitate the generation of high-quality action selection outputs, resulting in effective control of the agent in performing a given task.

[0102] The system can perform one iteration of process 300 for each batch of one or more transitions sampled from the replay buffer. At the start of each iteration, the system can generate an encoded representation of the current observation contained in each sampled transition by processing the current observation using the encoder subnetwork according to the current parameter values ​​of the encoder subnetwork. However, unlike at inference, where the encoded representation is directly given as input to the attention subnetwork, the system generates a masked encoded representation from the encoded representation and then gives that masked encoded representation as input to the attention subnetwork.

[0103] As mentioned above, the encoded representation of the current observation can take the form of an input vector having the respective input values ​​at each of the multiple input positions in the input order. In contrast, a masked encoded representation masks the respective input values ​​at each of the one or more input positions in the multiple input positions in the input order, that is, it includes a fixed value (e.g., negative infinity, positive infinity, or another predetermined mask value) in place of the original input value at each of the one or more input positions.

[0104] To generate a masked encoded representation (hereinafter referred to as the "masked input vector"), the system selects one or more input positions in the input order from the encoded representation and applies a mask to each input value at each of the selected input positions; that is, it replaces each input value at each of the selected input positions with a fixed value. For example, the selection may be performed via random sampling, and a fixed amount of input values ​​(e.g., 10%, 15%, or 20%) may be masked for each encoded representation.

[0105] The system uses an attention subnetwork to process a masked input vector that masks the respective input values ​​at one or more input positions in the input sequence, depending on the current parameter values ​​of the attention subnetwork, in order to generate predictions of the respective input values ​​at each input position in the input sequence (step 302). That is, during controlled learning training, the attention subnetwork is trained to perform an auxiliary task of reconstructing the input vector from its masked version.

[0106] The system evaluates the contrast learning objective function (step 304). The contrast learning objective function measures the contrast learning loss of the attention subnetwork (e.g., contrast loss 420 in Figure 4) when predicting the masked input value from processing the masked input vector.

[0107] Specifically, for each input position of one or more input positions among multiple input positions in the input order, the contrast learning objective function may measure a first difference between i) a prediction of each input value and ii) each input value in the input vector corresponding to the encoded representation of the current observation. The first difference may be called the difference evaluated with respect to “positive examples”. As illustrated in the embodiment of Figure 4, at a given time step 402A, the system can determine, for each input position, the respective difference between i) the attention subnetwork training output ("X1") 414A containing the prediction of each input value at that input position and ii) the masked input vector that masks each input value that was originally included in the encoded representation of the observation ("Y1") 412A corresponding to that given time step 402A.

[0108] Alternatively, the controlled learning objective function may measure a second difference between i) the prediction of each input value and ii) the prediction of each input value in another input vector corresponding to an encoded representation of the augmented current observation. The second difference may be called the difference evaluated with respect to the “negative example”. Furthermore, or alternatively, the second difference can be the difference between i) the prediction of each input value and ii) the prediction of each input value in another input vector generated by the attention subnetwork from the masked input vector corresponding to the augmented current observation. That is, the second difference can be the difference evaluated with respect to the attention subnetwork training output generated with respect to the augmented current observation. For example, the first and second differences can be evaluated in terms of Kullback-Leibler divergence.

[0109] In detail, contrast representation learning typically utilizes data augmentation techniques to create groups of data that can be compared in order to generate meaningful training signals.

[0110] In some implementations, the system can rely on the sequential nature of the input data, and the augmented present observation can be a future observation characterizing a future state of the environment that follows the current state. Alternatively, the augmented present observation can be a historical observation characterizing a past state of the environment that preceded the current state.

[0111] As illustrated in the embodiment of Figure 4, in a given time step 402A, the system can determine, for each input position, the difference between i) an attention subnetwork training output ("X1") 414A containing predictions of the respective input values ​​at that input position, and ii) the difference between the respective input values ​​in another input vector corresponding to an encoded representation 412B of a future observation to be received in a future time step 402B. Furthermore, or alternatively, the system can determine, for each input position, the difference between i) an attention subnetwork training output ("X1") 414A containing predictions of the respective input values ​​at that input position, and ii) an attention subnetwork training output ("X2") 414B containing predictions of the respective input values ​​in another input vector generated by the attention subnetwork from a masked input vector corresponding to a future time step 402B. Specifically, in these embodiments, each input value in the other input vector can have the same input position within the other input vector as each input value in the input vector corresponding to a sampled transition.

[0112] In some other implementations, the system may instead rely on visual representation-based augmentation techniques, where the augmented current observation can be, for example, a geometrically transformed or color-space transformed representation of the current observation.

[0113] The system determines to update the current parameter values ​​of the attention subnetwork based on calculating the gradient of the controlled learning loss with respect to the attention subnetwork parameters (step 306). Furthermore, the system determines to update the current parameter values ​​of the encoder subnetwork via backpropagation.

[0114] In some implementations, the system then proceeds to update the current parameter values ​​based on the gradient of the controlled learning loss by using conventional optimization algorithms, such as stochastic gradient descent, RMSprop, or Adam optimization algorithms, including Adam ("AdamW") optimization algorithms with weight decay. Alternatively, the system proceeds to update the current parameter values ​​only after steps 302–306 have been performed for the entire batch of sampled transitions. In other words, the system proceeds to update the current parameter values ​​based on the combined gradient by combining each gradient determined during a fixed number of iterations of steps 302–306 by calculating their weighted or unweighted mean.

[0115] The system can repeatedly execute steps 302-306 until the control learning training termination criteria are met, for example, after steps 302-306 have been executed a predetermined number of times, or after the gradient of the control learning objective function has converged to a specified value.

[0116] In some implementations, the system can optimize the reinforcement learning loss together with the control learning loss. Therefore, in these implementations, the system combines the reinforcement learning loss and the control learning loss, for example, by calculating their weighted sum, and then updates the current parameter values ​​based on the combined loss. In these implementations, steps 302-306 can be repeated until the system's RL training is complete, for example, after the gradient of the reinforcement learning objective function converges to a specified value.

[0117] Figure 5 shows a quantitative example of the performance gains that can be achieved by using the control neural network system described herein. Specifically, Figure 5 shows a list of scores (where higher scores indicate higher rewards) received by an agent controlled using the control neural network system 110 of Figure 1 for a range of DeepMind Lab tasks. As a platform designed for the research and development of general artificial intelligence and machine learning systems, DeepMind Lab (https: / / arxiv.org / abs / 1612.03801) can be used to study how autonomous artificial agents are capable of learning complex tasks in broad, partially observed, and visually diverse environments. As illustrated, it can be observed that the "coberl" agent (corresponding to an agent controlled using the control neural network system described herein) generally outperforms the "gtrxl" agent (corresponding to an agent controlled using an existing control system that uses only attention mechanisms, i.e., the "Gated Transformer XL" system described in Parisotto et al., Stabilizing transformers for reinforcement learning, arXiv:1910.06764) by a considerable margin for the majority of tasks.

[0118] This specification uses the term “configured” in relation to system components and computer program components. One or more computer systems being configured to perform a particular operation or action means that, when in operation, software, firmware, hardware, or a combination thereof is installed on the system that causes the system to perform that operation or action. One or more computer programs being configured to perform a particular operation or action means that, when executed by a data processing device, the program contains instructions that cause the device to perform that operation or action.

[0119] Embodiments of subject matter and functional operation described herein can be implemented in digital electronic circuits, in tangible computer software or computer firmware, or in computer hardware, including structures disclosed herein and their structural equivalents, or combinations thereof. Embodiments of subject matter described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transient storage medium to be executed by a data processing device or to control the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable memory board, a random-access memory device or a serial-access memory device, or a combination of one or more of these storage mediums. Alternatively, or further, the program instructions can be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiving device to be executed by a data processing device.

[0120] The term "data processing device" refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or multiple computers. The device may also be a dedicated logic circuit, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), or may further include such dedicated logic circuits. Optionally, in addition to hardware, the device may include code that constitutes an execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or a combination of one or more of these.

[0121] Computer programs, which may be called, or described as, programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as standalone programs suitable for use in a computing environment, or as modules, components, subroutines, or other units. A program may or may not correspond to a file in a file system. A program may be stored in a part of a file that holds other programs or other data, for example, in one or more scripts stored in a markup language document, in a single file dedicated to the program, or in a number of collaborative files, for example, in a file that stores one or more modules, subprograms, or parts of code. Computer programs can be deployed to run on one computer, or on a number of computers located in one site, or distributed across multiple sites and connected to each other by a data communication network.

[0122] In this specification, the term “database” is used broadly to refer to any collection of data, which does not need to be structured in any particular way, or not structured at all, and can be stored on a storage device in one or more locations. For example, an index database can contain many collections of data, each of which may be organized differently and accessed differently.

[0123] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or software components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and running on the same computer or on the same number of computers.

[0124] The processes and logic flows described herein can be executed by one or more programmable computers that run one or more computer programs to perform functions by manipulating input data and generating outputs. Alternatively, the processes and logic flows can be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0125] A computer suitable for running computer programs can be based on a general-purpose microprocessor, a dedicated microprocessor, both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random-access memory, or both. Essential elements of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by or incorporated into dedicated logic circuits. Generally, a computer is operationally coupled to one or more mass storage devices for storing data, including, for example, magnetic disks, magneto-optical disks, or optical disks, or to receive data from or transfer data to such mass storage devices, or both. However, a computer does not have to have such devices. Furthermore, computers can be embedded in other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, GPS receivers, or portable storage devices, such as Universal Serial Bus (USB) flash drives.

[0126] Computer-readable media suitable for storing computer program instructions and data include, for example, all forms of non-volatile memory, non-volatile media, and non-volatile memory devices, including semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0127] To enable interaction with the user, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) monitor or an LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, to which the user can provide input to the computer. Other types of devices can also be used to enable interaction with the user; for example, the feedback given to the user can be any form of perceptual feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user can be received in any form, including acoustic input, voice input, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from devices used by the user, for example, by sending web pages to a web browser on the user's device in response to a request received from that web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, for example, a smartphone running a messaging application, and receiving response messages from the user in return.

[0128] A data processing device for implementing a machine learning model may also include, for example, a dedicated hardware accelerator unit for handling the typical, computationally intensive portion of machine learning training or production, namely the inference workload.

[0129] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0130] Embodiments of the subject matter described herein can be implemented in a computing system including backend components, for example, as a data server; in a computing system including middleware components, for example, as an application server; or in a computing system including frontend components, for example, as a client computer having a graphical user interface, web browser, or app that allows a user to interact with an implementation of the subject matter described herein; or in a computing system including one or more such backend components, middleware components, or frontend components. The components of the system can be connected to each other by digital data communication in any form or medium, for example, by a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), for example, the Internet.

[0131] A computing system can include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The relationship between the client and the server arises thanks to computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device for the purpose of displaying data to a user interacting with a device acting as a client, and for the purpose of receiving user input from such a user. Data generated on the user device, such as the results of user interaction, can be received from the device by the server.

[0132] This specification includes many specific implementation details, which should not be interpreted as limitations on the scope of the invention or on the scope of the claims, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Furthermore, some features described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented separately or in any suitable partial combination in multiple embodiments. Moreover, features may be described in the preceding paragraph as acting in a certain combination, and may even be initially claimed as such, but one or more features from the claimed combination may, in some cases, be removed from that combination, and the claimed combination may be a partial combination or a variation of a partial combination.

[0133] Similarly, while the operations are depicted in the drawings in a specific order and described in the claims, this should not be understood as requiring that such operations must be performed in a specific illustrated order, or in a sequential order, or that all exemplified operations must be performed in order to achieve the desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems should generally be understood as being able to be integrated together as a single software product or packaged as a number of software products.

[0134] Specific embodiments of the subject matter have been described. Other embodiments are included in the appended claims. For example, the actions described in the claims can be performed in a different order, and it is still possible to achieve the desired results. As an example, the processes depicted in the appended drawings do not necessarily require a specific illustrated order or a sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0135] 100 Reinforcement Learning Systems 102 Agents 104 Environment 106 Actions 108 Observations 110 Control Neural Network Systems 118 Network Parameters 120 Action Selection Neural Network 122 Action Selection Output 124 Encoder Subnetwork 126 Encoded representation 128 Attention Subnetwork 130 Caution Subnetwork Output 132 Gating Subnetworks 134 Gating Subnetwork Output 136 Recurrent Subnetworks 140 Action Selection Subnetwork 150 training engines 402A, 402B Time Step 412A, 412B Encoded representation of observations 414A, 414B Attention Subnetwork Training Output 410 RL loss 420 Controlled Loss

Claims

1. A system for performing tasks, One or more computers, One or more storage devices that store instructions causing the one or more computers to implement a neural network configured to perform the task when executed by the one or more computers, Equipped with, The aforementioned neural network is An attention subnetwork is configured to receive, in each of a plurality of time steps, an attention subnetwork input generated from the input received by the neural network for the time step, and to generate an attention subnetwork output based on at least applying an attention mechanism to the attention subnetwork input, The recurrent subnetwork is configured to receive a recurrent subnetwork input generated from the attention subnetwork output and generate a recurrent subnetwork output in each of the plurality of time steps to update the current hidden state of the recurrent subnetwork corresponding to the time step, wherein the current hidden state is generated by processing one or more previous recurrent subnetwork inputs, and updating the current hidden state modifies the current hidden state by processing the received recurrent subnetwork input. In each of the plurality of time steps, an output subnetwork is configured to receive an output subnetwork input generated from the recurrent subnetwork output, and to process the output subnetwork input to generate an output for the task. A system that includes these features.

2. The neural network is The system according to claim 1, further comprising an encoder subnetwork configured to process the input received by the neural network for each of a plurality of time steps, and to generate an encoded representation of the input.

3. The system according to claim 2, wherein the attention subnetwork input includes the encoded representation of the input.

4. The system according to any one of claims 1 to 3, wherein the attention mechanism is a masked attention mechanism.

5. The system according to any one of claims 1 to 4, wherein the recurrent subnetwork comprises one or more long short-term memory (LSTM) layers.

6. The output for the task includes a numerical probability value for each output item in the set of possible output items, The system according to any one of claims 1 to 5, wherein performing the task includes selecting an output item for the time step from the set of possible output items based on the respective numerical probability values.

7. The system according to any one of claims 2 to 6, wherein the neural network further comprises a gating layer configured to apply a gating mechanism to i) an encoded representation of the input and ii) the attention subnetwork output, in order to generate the recurrent subnetwork input.

8. The system according to claim 7, wherein applying the gating mechanism to the encoded representation of the input and the attention subnetwork output comprises applying a gated recurrent unit (GRU) to the encoded representation of the input and the attention subnetwork output.

9. The system according to any one of claims 1 to 8, wherein in each of the plurality of time steps, the attention subnetwork input includes an encoded representation of the input and an encoded representation of one or more previous inputs received by the neural network for one or more previous time steps.

10. A method performed by one or more computers, The steps include: using an attention subnetwork of a neural network configured to perform a task, processing the attention subnetwork input generated from the input received by the neural network, and generating an attention subnetwork output based on applying an attention mechanism to the attention subnetwork input; A step of generating a recurrent subnetwork output by processing a recurrent subnetwork input generated from the attention subnetwork output in order to update the current hidden state of the neural network, wherein the current hidden state is generated by processing one or more previous recurrent subnetwork inputs, and updating the current hidden state modifies the current hidden state by processing the received recurrent subnetwork input. The steps include: processing the output subnetwork input generated from the recurrent subnetwork output using the output subnetwork of the neural network to generate an output for the task; Methods that include...

11. By the encoder subnetwork of the neural network, The method according to claim 10, further comprising the step of processing the input received by the neural network to generate an encoded representation of the input.

12. The method according to claim 11, wherein the attention subnetwork input includes the encoded representation of the input.

13. The method according to claim 10, wherein the attention mechanism is a masked attention mechanism.

14. The method according to claim 10, wherein the recurrent subnetwork comprises one or more long short-term memory (LSTM) layers.

15. The output for the task includes a numerical probability value for each output item in the set of possible output items, The method according to claim 10, wherein performing the task includes selecting an output item for a time step from the set of possible output items based on the respective numerical probability values.

16. The method according to claim 11, further comprising the step of applying a gating mechanism to i) an encoded representation of the input and ii) the attention subnetwork output, in order to generate the recurrent subnetwork input by a gating layer of the neural network.

17. The method according to claim 16, wherein applying the gating mechanism to the encoded representation of the input and ii) the attention subnetwork output comprises applying a gated recurrent unit (GRU) to the encoded representation of the input and ii) the attention subnetwork output.

18. The method according to claim 11, wherein the attention subnetwork input includes an encoded representation of the input and an encoded representation of one or more previous inputs received by the neural network.

19. One or more computer storage media storing instructions that, when executed by one or more computers, cause one or more computers to perform an operation of any one of the methods described in any one of claims 10 to 18.

Images