Attention neural network with short-term memory unit

Abstract

To provide a system for controlling an agent interacting with an environment to perform a task.SOLUTION: The system includes an action selection neural network configured to generate action selection outputs that are used to select actions to be performed by an agent. The action selection neural network includes: an encoder subnetwork configured to generate encoded representations of current observations; an attention subnetwork configured to generate attention subnetwork outputs using an attention mechanism; a recurrent subnetwork configured to generate recurrent subnetwork outputs; and an action selection subnetwork configured to generate action selection outputs that are used to select actions to be performed by the agent in response to the current observations.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 146,361, filed on February 5, 2021. The disclosure of the prior application is hereby incorporated by reference in its entirety and made a part of this application's disclosure.

[0002] This specification relates to reinforcement learning.

Background Art

[0003] In a reinforcement learning system, an agent interacts with the environment by receiving an observation that characterizes the current state of the environment and performing an action selected by the reinforcement learning system in response.

[0004]

[0005] Some reinforcement learning systems select an action to be performed by an agent in response to receiving a given observation according to the output of a neural network. A neural network is a machine learning model that uses one or more layers of non - linear units to predict an output for received inputs. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values of its respective set of parameters.

Prior Art Documents

Non - Patent Documents

[0006]

Non - Patent Document 1

Non - Patent Document 2

[0007] This specification generally describes a reinforcement learning system that controls an agent interacting with an environment.

[0008] The subject matter described in this specification can be implemented in certain embodiments so as to achieve one or more of the following advantages.

[0009] By incorporating the self-attention mechanism of an attention-based neural network with the memory mechanism of a recurrent neural network such as a long short-term memory (LSTM) neural network into an action selection neural network used by a reinforcement learning system to select actions to be performed by an agent, the techniques described can provide temporally structured information to the action selection neural network during training or after training, i.e., at runtime, to improve the quality of the action selection output. Specifically, the techniques described effectively utilize both a self-attention mechanism that extracts long-range dependencies to integrate information about an agent's past interactions with the environment at multiple different timescales and a memory mechanism that reasons about short-term dependencies, such that the action selection neural network can reason about events over multiple timescales and adjust the future action selection policy accordingly.

[0010] Furthermore, the techniques described herein optionally include an implementation of a trainable gating mechanism to enable the action selection neural network to more effectively combine information computed using an attention-based neural network and a recurrent neural network. This effective combination can be particularly advantageous in complex environment settings as it allows for greater flexibility in determining which information should be processed when controlling a robotic agent. Here, the term "gating mechanism" refers to a unit that forms a dataset based on both an input to a neural network and an output of the neural network. A trainable gating mechanism is a gating mechanism where the dataset is further based on one or more adjustable parameter values. The gating mechanism may be used, for example, to generate a dataset based on both an input to an attention subnetwork and an output of the attention subnetwork.

[0011] Therefore, the reinforcement learning system described in this specification can achieve performance superior to that of conventional reinforcement learning systems, for example, by receiving more cumulative external rewards, in controlling an agent to execute tasks. The reinforcement learning system described in this specification trains an action selection neural network faster than conventional reinforcement learning systems that do not utilize a self-attention mechanism, do not utilize a memory mechanism, or do not utilize either of them. Further, in addition to training the neural network to maximize the cumulative reward, by training the action selection neural network for a contrastive learning auxiliary task, the reinforcement learning system described in this specification can enhance the feedback signal received during the training of the action selection neural network to further improve the training, for example, by encouraging the learning of representations that assist in obstacle avoidance or trajectory planning. Therefore, the reinforcement learning system described in this specification enables more efficient use of computational resources in training.

[0012] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

[0014] Like reference numerals and names in the various drawings refer to like elements.

[0015] This specification describes a reinforcement learning system that controls an agent that interacts with an environment by processing data (i.e., "observations") that characterize the current state of the environment at each time step of a number of time steps to select an action to be performed by the agent at that time step.

[0016] At each time step, the state of the environment at that time step depends on the state of the environment at the previous time step and the action performed by the agent at the previous time step.

[0017] In some implementations, the environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment, such as a robot that moves through the environment or an autonomous or semi-autonomous land, air, or sea vehicle.

[0018] In these implementations, the observations may include, for example, one or more of an image that captures the observation as the agent interacts with the environment, object position data, and sensor data, such as sensor data from an image sensor, a distance sensor, or a position sensor, or from an actuator.

[0019] For example, in the case of a robot, the observations may include data characterizing the current state of the robot, such as joint positions, joint velocities, joint forces, torques or accelerations, such as gravity-compensated torque feedback, and / or one or more of the overall or relative postures of items held by the robot.

[0020] In the case of a robot or other mechanical agent, or a vehicle, the observations may similarly include one or more of position, linear or angular velocity, force, torque or acceleration, and the overall or relative posture of one or more parts of the agent. The observations may be defined in one, two, or three dimensions and may be absolute and / or relative observations.

[0021] Also, the observations may include sensed electronic signals such as motor current or temperature signals, and / or image data or video data from, for example, a camera or LIDAR sensor, such as data from the agent's sensors, or data from sensors placed separately from the agent in the environment.

[0022] In these implementations, the actions may be control inputs for controlling the robot, such as torques on the joints of the robot, or high-level control commands, or control inputs for controlling autonomous or semi-autonomous ground, air, or sea vehicles, such as torques on the vehicle's control surfaces or other control elements, or high-level control commands.

[0023] In other words, an action can include, for example, position data, velocity data, or force / torque / acceleration data regarding one or more joints of a robot, or one or more parts of another mechanical agent. The action data may further or alternatively include electronic control data such as motor control data, or more generally, data for controlling one or more electronic devices in an environment where the control affects the observed state of the environment. For example, in the case of an autonomous or semi-autonomous ground, air, or sea vehicle, the action may include actions for controlling the navigation of the vehicle, such as steering, and the movement, such as braking and / or accelerating.

[0024] In some other application forms, the agent may control actions in a real-world environment including items of equipment, for example, in a data center, in a power / water distribution system, or in a manufacturing plant or service facility. In that case, the observation may be related to the operation of the plant or facility. For example, the observation may include observing the power or water usage by the equipment, or observing power generation control or power distribution control, or observing the usage amount of resources or the calculation of waste. The action may include actions for controlling an item of equipment of the plant / facility or imposing operating conditions on an item of equipment of the plant / facility, and / or actions for causing a change in settings in the operation of the plant / facility, such as adjusting or turning on / off components of the plant / facility.

[0025] In the case of an electronic agent, observations may include data from one or more sensor monitoring parts of a plant or service facility, such as current sensors, voltage sensors, power sensors, temperature sensors, and other sensors, and / or electronic signals representing the functions of electronic items and / or mechanical items of the equipment. For example, the real-world environment may be a manufacturing plant or a service facility, the observations may be related to the operation of the plant or facility, such as the resource usage, e.g., power consumption, and the agent may control actions or operations in the plant / facility, e.g., to reduce the resource usage. In some other implementations, the real-world environment may be a renewable energy plant, the observations may be related to the operation of the plant, e.g., to maximize the current or planned future power generation, and the agent may control actions or operations in the plant to achieve this.

[0026] As another example, the environment may be a chemical synthesis environment or a protein folding environment such that each state is a state of a protein chain or of each of one or more intermediate or precursor chemical substances, and the agent is a computer system for determining how to fold the protein chain or how to synthesize the chemical substances. In this example, the actions are possible folding actions for folding the protein chain or actions for assembling precursor chemical substances / intermediates, and the results to be achieved may include, for example, folding the protein such that the protein is stable and the protein achieves a specific biological function, or resulting in a reasonable synthesis route for the chemical substances. As another example, the agent may be a mechanical agent that executes or controls protein folding actions or chemical synthesis steps automatically selected by the system without human interaction. The observations may include direct or indirect observations of the state of the protein or chemical substance / intermediate / precursor and / or may be derived from simulations.

[0027] In some implementations, the environment may be a simulated environment, and the agent may be implemented as one or more computers that interact with the simulated environment.

[0028] The simulated environment may be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent may be a simulated vehicle that moves within the motion simulation. In these implementations, the action may be a simulated user or a control input that controls the simulated vehicle.

[0029] In some implementations, the simulated environment may be a simulation of a particular real-world environment. For example, the system may be used to select actions in a simulated environment during training or evaluation of a control neural network, and after training or evaluation, or both, may be deployed to control a real-world agent in the real-world environment simulated by the simulated environment. This can avoid unnecessary wear and tear, and damage to, the real-world environment or real-world agent, and enable the control neural network to be trained and evaluated against situations that rarely occur or are difficult to reproduce in the real-world environment.

[0030] Generally, in cases of simulated environments, the observation may include the observations described above, or a simulated version of one or more of the observations of the types described above, and the action may include the actions described above, or a simulated version of one or more of the actions of the types described above.

[0031] As an option, in any of the previous implementation forms, the observations at any given time step may include data from previous time steps that may be useful for characterizing the environment, such as actions performed at the previous time step, rewards received at the previous time step, and the like.

[0032] FIG. 1 shows an exemplary reinforcement learning system 100. The reinforcement learning system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations where the systems, components, and techniques described later are implemented.

[0033] The system 100 controls the agent 102 to interact with the environment 104 by, for example, selecting the action 106 to be performed by the agent 102 and then sending control data to the agent 102 that instructs the agent 102 to perform the action 106. In some cases, the reinforcement learning system 100 may be implemented on the agent 102 or may be a component of the agent 102, and the control data is sent to the (multiple) actuators of the agent.

[0034] The execution of the selected action 106 by the agent 102 generally causes the environment 104 to transition to successive new states. By repeatedly causing the agent 102 to operate in the environment 104, the system 100 can control the agent 102 to complete the specified task.

[0035] The system 100 includes a control neural network system 110 and one or more memories that store a set of model parameters 118 (``network parameters'') of the neural network included in the control neural network system 110.

[0036] At a high level, the control neural network system 110 is configured to process an input including a current observation 108 characterizing the current state of the environment 104 in each time step of a number of time steps, in response to the current observation 108, to generate an action selection output 122 that can be used to select a current action 106 to be executed by the agent 102 according to the model parameters 118.

[0037] The control neural network system 110 includes an action selection neural network 120. The action selection neural network 120 is implemented with a neural network architecture that enables the system to detect events occurring at different time scales and react to those events, thereby improving the quality of the action selection output 122. Specifically, the action selection neural network 120 includes an encoder sub-network 124, an attention sub-network 128, a gating sub-network 132 (preferably), a recurrent sub-network 136, and an action selection sub-network 140. Each sub-network can be implemented as a group of one or more neural network layers in the action selection neural network 120.

[0038] In each time step of a number of time steps, the encoder sub-network 124 receives an encoder sub-network input including a current observation 108 characterizing the current state of the environment 104, and an encoded representation of the current observation 108 ( "Y" tTo generate 126, it is configured to process the encoder subnetwork input according to the trained parameter values of the encoder subnetwork. The encoded representation 126 can be in the form of an ordered collection of numerical values, for example, a vector or matrix of numerical values. For example, the encoded representation 126 subsequently given as input to the attention subnetwork 128 can be an input vector having respective input values at each of a number of input positions in the input order. In some implementations, the encoded representation 126 has the same dimension as the observation 108, and in some other implementations, the encoded representation 126 has a smaller dimension compared to the observation 108 for reasons of computational efficiency. In some implementations, in addition to giving the encoded representation 126 as input to the attention subnetwork 128, the system also stores the encoded representation 126 generated at a given time step in a memory buffer or lookup table so that the encoded representation 126 can be used, for example, in future time steps following that given time step.

[0039] When the observation is an image, the encoder subnetwork 124 can be a convolutional subnetwork configured to process the observation with respect to time steps, for example, a convolutional neural network having residual blocks. In some cases, for example, when the observation contains lower-dimensional data, the encoder subnetwork 124 can further or alternatively include one or more fully-connected neural network layers.

[0040] Note that the attention sub-network 128 is a network that includes one or more attention neural network layers. Each attention layer operates on each input sequence that includes respective input vectors at each position of one or more positions (e.g., a plurality of concatenated input vectors). At each time step of a number of time steps, the attention sub-network 128 receives an attention sub-network input that includes an encoded representation 126 of the current observation 108 and an encoded representation of one or more previous observations, and at least in part, applies an attention mechanism to each encoded representation of the current observation and one or more previous observations, and processes the attention sub-network input according to the trained parameter values of the attention sub-network so as to generate an attention sub-network output (“X t ”). That is, the attention sub-network output (“X t ”) 130 is an output determined or otherwise derived from an updated (i.e., “attention-directed”) representation of the encoded representation generated by using one or more attention neural network layers.

[0041] Specifically, in addition to the encoded representation 126 of the current observation 108, the attention sub-network input also includes an encoded representation of one or more previous observations that characterize one or more previous states of the environment 108 immediately preceding the current state of the environment. Each encoded representation of a previous observation can be in the form of a respective input vector that includes respective input values at each input position of a number of input positions in the input order. Thus, the attention sub-network input can be a concatenated input vector consisting of a number of individual input vectors, each corresponding to an encoded representation of an observation of a different previous state of the environment 108 (including the current state) up to the current state.

[0042] Generally, the attention layer within the attention subnet 128 can be configured in any of a variety of configurations. For details of embodiments of the configuration of the attention layer and other components of the attention subnet, reference is made in its entirety to Vaswani et al., Attention Is All You Need, arXiv:1706.03762, and Parisotto et al., Stabilizing transformers for reinforcement learning, arXiv:1910.06764, which are incorporated herein by reference and described in more detail therein. For example, the attention mechanism applied by the attention layer within the attention subnet 128 can be a self-attention mechanism, such as a multi-head self-attention mechanism.

[0043] Generally, an attention mechanism maps a set of queries and key-value pairs to an output, where the queries, keys, and values are all vectors derived from the inputs to the attention mechanism based on their respective matrices. The output is calculated as a weighted sum of the values, where the weight assigned to each value is calculated by a fitness function of the query having the corresponding key, such as an inner product or a scaled dot product. Generally, an attention mechanism determines the relationship between two sequences, while a self-attention mechanism is configured to relate different positions within the same sequence and determine a transformed version of that sequence as the output. The attention layer input may include vectors for each element of the input sequence. These vectors are used by the self-attention mechanism to provide an input and determine a new representation of the same sequence for the attention layer output that also includes vectors for each element of the input sequence. The output of the self-attention mechanism may be used as the attention layer output.

[0044] In some implementations, the attention mechanism is, for example, a query transformation defined by a matrix W Q a key transformation defined by a matrix W K and, for example, a matrix W VEach value transformation defined by is used to determine, for each input vector X of the input sequence, a sequence to which attention for output is directed, using each query vector Q = XW Q and key vector K = XW K and value vector V = XW V configured to be applied to derive. For example, the attention mechanism may be an inner product attention mechanism applied by applying each query vector to each key vector to determine respective weights for each value vector, and then combining the value vectors using the respective weights to determine the attention layer output for each element of the input sequence. The attention layer output may be scaled by a scaling factor, such as the square root of the dimensions of the query and key, to implement scaled dot-product attention. Thus, for example, the output of the attention mechanism may be

[0045]

Number

[0046] determined as , where d is the dimension of the key (and value) vector. In another implementation, the attention mechanism comprises an "additive attention" mechanism that calculates a fitness function using a feed-forward network having a hidden layer.

[0047] The attention mechanism may implement multi-head attention, i.e., the attention mechanism may apply a number of different attention mechanisms in parallel. These outputs may then be combined, if necessary, with a learned linear transformation applied to reduce to the original dimension, e.g., concatenated.

[0048] Note that a neural network layer or a self-attention neural network layer is a neural network layer that includes an attention mechanism or a self-attention mechanism (which operates on an attention layer input to generate an attention layer output). The attention sub-network 128 may comprise a single attention layer or, alternatively, may comprise a sequence of attention layers where each attention layer, other than the first attention layer, receives as input the output from a preceding attention layer in the sequence.

[0049] Furthermore, in this embodiment, the self-attention mechanism can be masked such that any given position in the input sequence does not direct attention to any position after that given position in the input sequence. For example, for subsequent positions after that given position in the input sequence, the attention weights for the subsequent positions are set to 0.

[0050] The recurrent sub-network 136 is a network that includes one or more recurrent neural network layers, such as one or more long short-term memory (LSTM) layers, one or more gated recurrent unit (GRU) layers, one or more vanilla recurrent neural network (RNN) layers, etc. The recurrent sub-network 136 is configured to, at each time step of a number of time steps, receive a recurrent sub-network input and process it according to the trained parameter values of the recurrent sub-network to update the current hidden state of the recurrent sub-network corresponding to that time step so as to generate a recurrent sub-network output.

[0051] Specifically, the recurrent subnetwork input is derived from the attention subnetwork output 130. In some implementations, the action selection neural network 120 can directly provide the attention subnetwork output 130 as an input to the recurrent subnetwork 136. Alternatively, in some implementations, the action selection neural network 120 can utilize a gating subnetwork 132 to combine the output 126 of the encoder subnetwork 124 and the output 130 of the attention subnetwork 128. Thus, the recurrent subnetwork input can be the output (「Z t 」) 134 of the gating subnetwork 132 of the action selection neural network 120.

[0052] In some of these implementations, the gating mechanism implemented by the gating subnetwork 132 is a fixed, addition (or concatenation) mechanism, and the gating subnetwork can include an addition (or concatenation) layer configured to receive the encoded representation 126 of the current observation and the attention subnetwork output 130, and to generate an output based on the representation 126 and the output 130. For example, the gating subnetwork can calculate the addition (or concatenation) of i) the encoded representation 126 of the current observation and ii) the attention subnetwork output 130 along a predetermined dimension of the received layer input.

[0053] In some of these other implementations, the gating mechanism implemented by the gating subnet network 132 is a learned gating mechanism that facilitates a more effective combination of information included in or otherwise derivable from the output 126 of the encoder subnet network 124 and the output 130 of the attention subnet network 128. In these implementations, the gating subnet network 132 is then configured to apply a learned GRU gating mechanism, according to the trained parameter values of the GRU layer, to i) the encoded representation 126 of the current observation and ii) the attention subnet network output 130 to generate a gated output, i.e., the gating subnet network output 134, which is then provided as an input to the recurrent subnet network 136. The combination of the encoded representation 126 and the attention subnet network output 130 using the GRU gating mechanism will be further described later with reference to FIG. 2. In some implementations, a skip (“residual”) connection can be configured between the encoder subnet network 124 and the gating subnet network 132, and the gating subnet network 132 can be configured to receive the encoded representation 126 via the skip connection in addition to directly receiving the attention subnet network output 130 from the attention subnet network 128.

[0054] The action selection sub-network 140 is configured to receive an action selection sub-network input and process the action selection sub-network input according to the trained parameter values of the action selection sub-network 140 to generate an action selection output 122 at each of a number of time steps. The action selection sub-network input includes a recurrent sub-network output and, in some implementations, further includes an encoded representation 126 generated by the encoder sub-network 124. In implementations where the action selection sub-network input also includes the encoded representation 126, a skip connection can be configured between the encoder sub-network 124 and the action selection sub-network 140, and the action selection sub-network 140 can be configured to receive the encoded representation 126 via the skip connection in addition to directly receiving the recurrent sub-network output from the recurrent sub-network 136.

[0055] Next, the system 100 uses the action selection output to select the action to be performed by the agent at the current time step. Some examples of using the action selection output to select the action to be performed by the agent are described next.

[0056] In one example, the action selection output 122 may include respective numerical probability values for each action in a set of possible actions that can be performed by the agent. The system can select the action to be performed by the agent, for example, by sampling an action according to the probability value for the action or by selecting the action with the highest probability value.

[0057] In another example, the action selection output 122 may directly define the action to be performed by the agent, for example, by defining the value of the torque to be applied to the joints of a robotic agent.

[0058] In another embodiment, the action selection output 122 may include respective Q-values for each action in a set of possible actions that can be performed by the agent. The system can be used to select the action to be performed by the agent (as described in the previous paragraph), and it is possible to process the Q-values to generate respective probability values for each possible action (e.g., using the softmax function). Also, the system can select the action with the highest Q-value as the action to be performed by the agent.

[0059] The Q-value for an action is an estimated value of the "return" resulting from the agent performing the action in response to the current observation and then selecting future actions to be performed by the agent according to the current values of the policy neural network parameters.

[0060] Return refers to a cumulative measure of the "reward" received by the agent, e.g., the time-discounted sum of rewards. The agent can receive respective rewards at each time step, where the reward is defined by a scalar numerical value and, for example, characterizes the agent's progress towards completing an assigned task.

[0061] In some cases, the system can select the action to be performed by the agent according to an exploration policy. For example, the exploration policy can be an ∈-greedy exploration policy where the system selects the action to be performed by the agent according to the action selection output 122 with probability 1 - ∈ and randomly selects an action with probability ∈. In this embodiment, ∈ is a scalar value within the range from 0 to 1.

[0062] The control of the agent using the control neural network system 110 will be described in more detail later with reference to FIG. 2.

[0063] To more effectively control the agent 102 that interacts with the environment 104, the reinforcement learning system 100 can use a training engine 150 to train the action selection neural network 120 to determine the trained values of the parameters 118 of the action selection neural network 120.

[0064] The training engine 150 is configured to train the action selection neural network 120 by repeatedly updating the model parameters 118, that is, the parameter values of the encoder sub-network 124, the attention sub-network 128, the gating sub-network 132, the recurrent sub-network 136, and the action selection sub-network 140, based on the interaction of the agent 102 (or another agent) with the environment 108 (or another instance of the environment).

[0065] Specifically, the training engine 150 trains the action selection neural network 120 through reinforcement learning and, additionally, optionally, through contrastive representation learning. Contrastive representation learning means that when the action selection neural network successively receives pairs of similar inputs (e.g., measured by a similarity measure, e.g., a distance measure such as the Euclidean distance), it generates outputs that are more similar to each other compared to the outputs generated by the action selection network from inputs that are further apart compared to the pairs of similar inputs. When receiving each input, the neural network components of the action selection neural network, particularly the attention subnetwork 128, are trained to generate outputs. In the examples given later, contrastive representation learning receives, as input, a masked form of the data generated by the encoder subnetwork 124 at a given time step, and generates an output that reconstructs the same data as generated by the encoder subnetwork 124 and / or trains the action subnetwork 128 to generate the data generated by the encoder subnetwork 124 at another time step.

[0066] In reinforcement learning, the action selection neural network 120 is trained based on the agent's interaction with the environment to optimize an appropriate reinforcement learning objective function. The architecture of the action selection neural network 120 is agnostic with respect to the choice of a strict RL training algorithm, and thus, the RL training can be on-policy (e.g., one of the RL algorithms more detailed in Song et al., V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control, arXiv:1909.12238) or off-policy (e.g., one of the RL algorithms more detailed in Kapturowski et al., Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018).

[0067] As part of this RL training process, the observations 108 received by the reinforcement learning system 100 during the interaction are encoded into an encoded representation from which the action selection output 122 is generated. Thus, learning to generate an encoded representation that provides useful information is an important factor for the success of the RL training. To that end, the training engine 150 evaluates a temporal contrast learning objective and uses it as a proxy supervision signal for training the attention subnetwork 128 to predict the masked portion of the attention subnetwork input, i.e., for masked prediction training of the attention subnetwork 128. Such a signal aims to learn a self-attention-consistent representation that contains appropriate information so that the action selection subnetwork 140 can effectively incorporate the information previously observed (or extracted) by the attention subnetwork 128.

[0068] In some implementations, by using contrastive learning techniques to assist in determining parameter value updates, the training engine 150 can, for example, improve the efficiency of the training process in terms of the amount of computational resources or real time consumed by the training process required to train the action selection neural network 120 to achieve or exceed the performance of the state of the art in controlling an agent to perform a given task.

[0069] Training the action selection neural network 120 will be described in more detail later with reference to FIGS. 3 and 4.

[0070] FIG. 2 is a flowchart of an exemplary process 200 for controlling an agent. For convenience, process 200 is described as being executed by one or more computer systems located at one or more locations. For example, a suitably programmed reinforcement learning system, such as the reinforcement learning system 100 of FIG. 1, can execute process 200.

[0071] Generally, the system can repeatedly execute process 200 to select each action (hereinafter referred to as the "current" action) to be performed by the agent in each state of the environment corresponding to that time step (hereinafter referred to as the "current" state) at each time step of a number of time steps.

[0072] The system receives a current observation characterizing the current state of the environment at the current time step and generates an encoded representation of the current observation by using an encoder subnetwork (step 202). For example, the current observation can include an image, a video frame, an audio data segment, a sentence in natural language, or the like. In some of these examples, the observation can also include information derived from the previous time step, such as the previous action executed, the reward received at the previous time step, or both. The encoded representation of the observation can be represented as an ordered collection of numerical values, such as a vector or matrix of numerical values.

[0073] The system processes an attention subnetwork input that includes the encoded representation of the current observation and the encoded representations of one or more previous observations by using an attention subnetwork to generate an attention subnetwork output (step 204). The one or more previous observations characterize one or more previous states of the environment that preceded the current state of the environment, and thus, the encoded representations of the one or more previous observations can be encoded representations generated by using the encoder subnetwork at one or more time steps preceding the current time step.

[0074] The attention subnetwork includes one or more attention neural network layers and is a neural network configured to generate an attention subnetwork output by applying an attention mechanism, such as a self-attention mechanism, at least in part, to the encoded representation of the current observation and the encoded representations of one or more previous observations characterizing one or more previous states of the environment. This use of the attention mechanism facilitates, for example, connecting long-distance data dependencies across observations of a long sequence of different states of the environment.

[0075] More specifically, the attention subnet input can include a concatenated input vector consisting of a number of individual input vectors corresponding to the respective encoded representations of the respective input values at each of a number of input positions in the input order. To generate the attention subnet output, each attention layer included in the attention subnet receives an attention layer input (which may likewise be in vector format) for each position of one or more layer input positions, and, for each respective specific layer input position in the layer input order, applies an attention mechanism to the attention layer input at that specific layer input position using one or more queries derived from the attention layer input at that specific layer input position to generate the respective attention layer output for that specific layer input position.

[0076] The system generates a combination of i) the encoded representation of the current observation and ii) the attention subnet output, and then provides that combination as an input to a recurrent subnet. To generate that combination, the system can utilize a gating subnet configured to apply a gating mechanism to i) the encoded representation of the current observation and ii) the attention subnet output to generate a recurrent subnet input. For example, the gating subnet can include a gated recurrent unit (GRU) layer configurable to apply a GRU gating mechanism of lower complexity compared to an LSTM layer by utilizing a smaller number of layer parameters. As another example, the gating subnet can also calculate an addition or concatenation of i) the encoded representation of the current observation and ii) the attention subnet output.

[0077] Specifically, in the previous embodiment, the GRU layer provides performance similar to that of a long short-term memory (LSTM) layer with a forget gate, but lacks an output gate, so it is a recurrent neural network layer with fewer parameters compared to the LSTM. In some implementations, this gating mechanism can be adapted as an update of the GRU layer unwound with respect to the depth of the action selection neural network instead of being unwound with respect to time. This means that while the GRU layer is a recurrent neural network (RNN) layer, the gating mechanism can use the same formula that the GRU layer uses to update the hidden state of the GRU layer over time to instead generate an "updated" combination of the inputs received in the gating subnetwork of the action selection neural network.

[0078] In these implementations, the GRU layer has a reset gate r and an update gate z, i.e.,

[0079]

Number

[0080] to calculate each, a non-linear function such as the sigmoid activation σ() is applied to the weighted combination of the received layer inputs, i.e., the encoded representation Y t and the attention subnetwork output X t to obtain the updated hidden state

[0081]

Number

[0082] i.e.,

[0083]

Number

[0084] To generate, a non-linear function such as the tanh activation tanh() is applied to the weighted combination of the encoded representation Y t and the element-wise product between the reset gate r and the attention subnetwork output X t where,

[0085] [Number]

[0086] is a weight (or bias) determined from the values of the GRU layer parameters,

[0087] [Number]

[0088] represents element-wise multiplication. Next, the GRU layer generates the gated output g (l) (x,y) (which can be used as the recurrent subnetwork input) as follows:

[0089] [Number]

[0090] generate.

[0091] The system processes the recurrent subnetwork input by using a recurrent subnetwork to generate a recurrent subnetwork output (step 206). The recurrent subnetwork is configured to receive the recurrent subnetwork input and update the current hidden state of the recurrent subnetwork by processing the received input, i.e., by processing the current received recurrent subnetwork input with the current hidden state of the recurrent subnetwork that was generated by processing the previous recurrent subnetwork input. The updated hidden state of the recurrent subnetwork after processing the recurrent subnetwork input is hereinafter referred to as the hidden state corresponding to the current time step. Once the updated hidden state corresponding to the current time step is generated, the system can use the updated hidden state of the recurrent subnetwork to generate a recurrent subnetwork output.

[0092] For example, the recurrent subnetwork can be a recurrent neural network that includes one or more long short-term memory (LSTM) layers. Because the LSTM layers are sequential in nature, the LSTM layers can effectively capture short-range dependencies, e.g., over successive observations of the most recent state of the environment.

[0093] The system uses an action selection subnetwork to process an action selection subnetwork input that includes a recurrent subnetwork output to generate an action selection output used to select an action to be performed by an agent in response to a current observation (step 208). In some implementations, the action selection subnetwork input also includes an encoded representation. In these implementations, the system is capable of computing a concatenation of i) the recurrent subnetwork output and ii) the encoded representation generated by the encoder subnetwork at the current time step to generate the action selection subnetwork input.

[0094] Next, the system can cause the agent to execute the selected action, i.e., instruct the agent to execute the action or pass a control signal to a control system regarding the agent to cause it to do so.

[0095] As described above, the components of the system can be trained through reinforcement learning combined with contrastive representation learning. In some implementations, the system maintains a replay buffer to assist with training. The replay buffer stores a number of transitions generated as a result of the agent interacting with the environment. Each transition represents information about the agent's interaction with the environment.

[0096] In these implementations, each transition is an experience tuple that includes i) a current observation characterizing the current state of the environment at a point in time, ii) a current action executed by the agent in response to the current observation, iii) a next observation characterizing the next state of the environment after the agent has executed the current action, i.e., the state of the environment resulting from the transition after the agent has executed the current action, and iv) a reward received in response to the agent executing the current action.

[0097] Briefly stated, in these implementation forms, RL training can involve repeatedly sampling a batch of one or more transitions from a replay buffer and then training an action selection neural network on the sampled transitions using an appropriate reinforcement learning algorithm. During each RL training iteration, the system can use the action selection neural network with the current parameter values of the action selection neural network to process the current observations included in each sampled transition to generate an action selection output, determine a reinforcement learning loss based on the action selection output, and then determine an update to the current values of the action selection network parameters based on calculating the gradient of the reinforcement learning loss with respect to the action selection neural network parameters.

[0098] Figure 4 is a diagram illustrating determining an update to the parameter values of the attention selection neural network. As shown, it is possible for the system to determine, at each time step, for example, each reinforcement learning loss regarding the action selection output generated at time step 402A, e.g., RL loss 410A.

[0099] Contrastive representation learning, which can be used to assist RL training to improve training data efficiency, will be further described later.

[0100] Figure 3 is a flowchart of an exemplary process 300 for determining an update to the parameter values of the attention selection neural network. For convenience, process 300 is described as being executed by a system of one or more computers located at one or more locations. For example, an appropriately programmed reinforcement learning system, e.g., the reinforcement learning system 100 of FIG. 1, can execute process 300.

[0101] Specifically, the system is capable of repeatedly executing process 300 to train an encoder of an action selection neural network and an attention subnetwork to generate, respectively, a high-quality encoded representation that facilitates the generation of high-quality action selection outputs that will result in effective control of the agent in performing a given task (e.g., providing useful information, or being predictive, or both), and an attention subnetwork output.

[0102] The system is capable of executing one iteration of process 300 for each batch of one or more transitions sampled from the replay buffer. At the start of each iteration, the system can generate an encoded representation of the current observation included in each sampled transition by using the encoder subnetwork to process the current observation according to the current parameter values of the encoder subnetwork. However, unlike at inference time when the encoded representation is directly given as input to the attention subnetwork, the system generates a masked encoded representation from the encoded representation and then gives the masked encoded representation as input to the attention subnetwork.

[0103] As described above, the encoded representation of the current observation can be in the form of an input vector having respective input values at each of a plurality of input positions in the input order. In contrast, the masked encoded representation masks the respective input values at one or more of the plurality of input positions in the input order, i.e., includes a fixed value (e.g., negative infinity, positive infinity, or another predetermined mask value) instead of the original input value at each of the one or more input positions.

[0104] To generate a masked encoded representation (hereinafter referred to as "masked input vector"), the system selects one or more of a plurality of input positions in the input order from the encoded representation, and applies a mask to each input value at each of the selected one or more input positions among the plurality of input positions in the input order, that is, replaces each input value with a fixed value at each of the selected input positions. For example, the selection may be performed via random sampling, and a fixed amount (e.g., 10%, 15%, or 20%) of the input values may be masked for each encoded representation.

[0105] The system uses an attention subnetwork to generate a prediction of each input value at each of one or more input positions among the plurality of input positions in the input order according to the current parameter values of the attention subnetwork, and processes the masked input vector that masks each input value at each of one or more input positions among the plurality of input positions in the input order (step 302). That is, during contrastive learning training, the attention subnetwork is trained to perform an auxiliary task of reconstructing the input vector from its masked version.

[0106] The system evaluates a contrastive learning objective function (step 304). The contrastive learning objective function measures the contrastive learning loss (e.g., the contrastive loss 420 in FIG. 4) of the attention subnetwork when predicting the masked input values from processing the masked input vector.

[0107] Specifically, for each of one or more input positions among a plurality of input positions in the input order, the contrastive learning objective function may measure a first difference between i) the prediction of each input value and ii) each input value in the input vector corresponding to the encoded representation of the current observation. The first difference may be referred to as the difference evaluated with respect to a "positive example". As illustrated in the embodiment of FIG. 4, at a given time step 402A, for each input position, the system can determine, for each input position, the respective difference between i) the attention subnetwork training output ("X1") 414A including the prediction of each input value at that input position and ii) the masked input vector that masks each input value initially included in the encoded representation ("Y1") 412A of the observation corresponding to that given time step 402A.

[0108] Also, the contrastive learning objective function may measure a second difference between i) the prediction of each input value and ii) each input value in another input vector corresponding to the enhanced encoded representation of the current observation. The second difference may be referred to as the difference evaluated with respect to a "negative example". Further, or alternatively, the second difference can be the difference between i) the prediction of each input value and ii) the prediction of each input value in another input vector generated by the attention subnetwork from the masked input vector corresponding to the enhanced current observation. That is, the second difference can be the difference evaluated with respect to the attention subnetwork training output generated with respect to the enhanced current observation. For example, the first difference and the second difference can be evaluated in terms of the Kullback-Leibler divergence.

[0109] Specifically, contrastive representation learning typically utilizes data augmentation techniques to create groupings of data that can be compared to generate a meaningful training signal.

[0110] In some implementations, the system can rely on the sequential nature of the input data, and the enhanced current observation can be a future observation that characterizes a future state of the environment that is after the current state. Additionally, or alternatively, the enhanced current observation can be a history observation that characterizes a past state of the environment that precedes the current state.

[0111] As illustrated in the embodiment of FIG. 4, at a given time step 402A, the system can, for each input position, i) determine an attention subnetwork training output ("X1") 414A that includes a prediction of each input value at that input position, and ii) determine each difference between each input value in another input vector corresponding to an encoded representation 412B of a future observation received at a future time step 402B. Additionally, or alternatively, the system can, for each input position, i) determine an attention subnetwork training output ("X1") 414A that includes a prediction of each input value at that input position, and ii) determine each difference between each input value in an attention subnetwork training output ("X2") 414B that includes a prediction of each input value in another input vector generated by the attention subnetwork from a masked input vector corresponding to the future time step 402B. Specifically, in these embodiments, each input value in the other input vector can have the same input position as each input value in the input vector corresponding to the sampled transition within the other input vector.

[0112] In some other implementations, the system can rely on techniques alternative to vision-based enhancement techniques, and the enhanced current observation can be, for example, a geometrically transformed representation or a color space-transformed representation of the current observation.

[0113] The system determines an update to the current parameter values of the attention subnetwork based on calculating the gradient of the contrastive learning loss with respect to the attention subnetwork parameters (step 306). Further, the system determines an update to the current parameter values of the encoder subnetwork via backpropagation of errors.

[0114] In some implementations, the system then proceeds to update the current parameter values based on the gradient of the contrastive learning loss by using a conventional optimization algorithm, such as Stochastic Gradient Descent, RMSprop, or an Adam optimization algorithm including the Adam (AdamW) optimization algorithm with weight decay. Alternatively, the system proceeds only to update the current parameter values when steps 302-306 have been executed for an entire batch of sampled transitions. In other words, the system combines each gradient determined during a fixed number of iterations of steps 302-306 by calculating their weighted average, or unweighted average, and proceeds to update the current parameter values based on the combined gradient.

[0115] The system can repeatedly execute steps 302-306 until a contrastive learning training termination criterion is met, e.g., after steps 302-306 have been executed a predetermined number of times, or after the gradient of the contrastive learning objective function has converged to a specified value.

[0116] In some implementations, the system can jointly optimize the reinforcement learning loss together with the contrastive learning loss. For this reason, in these implementations, the system, for example, combines the reinforcement learning loss and the contrastive learning loss by calculating their weighted sum, and then proceeds to update the current parameter values based on the combined loss. In these implementations, steps 302-306 can be repeatedly executed until the RL training of the system is complete, e.g., after the gradient of the reinforcement learning objective function has converged to a specified value.

[0117] Figure 5 shows a quantitative example of the performance gain that can be achieved by using the control neural network system described herein. Specifically, Figure 5 shows a list of scores received by an agent controlled using the control neural network system 110 of Figure 1 for a range of DeepMind Lab tasks (where a higher score indicates a greater reward). As a platform designed for the research and development of general artificial intelligence systems and machine learning systems, DeepMind Lab (https: / / arxiv.org / abs / 1612.03801) can be used to study how autonomous artificial agents can learn complex tasks in a wide, partially observable, and visually diverse environment. As shown, the "coberl" agent (corresponding to an agent controlled using the control neural network system described herein) generally outperforms the "gtrxl" agent (corresponding to an agent controlled using an existing control system that uses only an attention mechanism, i.e., the "Gated Transformer XL" system described in Parisotto et al., Stabilizing transformers for reinforcement learning, arXiv:1910.06764) by a significant margin for the majority of tasks.

[0118] This specification uses the term "configured" in relation to system components and computer program components. A system of one or more computers being configured to perform a particular operation or action means that, at runtime, software, firmware, hardware, or a combination thereof that causes the system to perform that operation or action is installed on the system. One or more computer programs being configured to perform a particular operation or action means that, when the one or more programs are executed by a data processing apparatus, the apparatus includes instructions that cause the apparatus to perform that operation or action.

[0119] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly realized computer software or computer firmware, or in computer hardware, including the structures disclosed in this specification and their structural equivalents, or combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random access memory device, or a serial access memory device, or a combination of one or more of these storage media. Alternatively, or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device for execution by a data processing apparatus.

[0120] The term "data processing apparatus" refers to data processing hardware and includes, by way of example, all kinds of apparatus, devices, and machines for processing data, including programmable processors, computers, or multiple processors or multiple computers. The apparatus can also be, or further include, dedicated logic circuitry, such as, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Optionally, in addition to the hardware, the apparatus can include code for creating an execution environment for computer programs, such as, for example, processor firmware, a protocol stack, a database management system, an operating system, or code constituting a combination of one or more of these.

[0121] A computer program, which may be referred to as or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including a compiled or interpreted language, or a declarative or procedural language, and can be deployed in any form, including as a stand-alone program suitable for use in a computing environment or as a module, component, subroutine, or other unit. The program may or may not correspond to a file in a file system. The program may be stored as part of a file that holds another program or other data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program, or in a number of cooperating files, such as files that hold one or more modules, subprograms, or portions of code. The computer program can be deployed to be executed on one computer or on a number of computers arranged at one site or distributed across a number of sites and interconnected by a data communication network.

[0122] As used herein, the term "database" is used broadly to refer to any collection of data, which need not be structured in any particular manner or at all and can be stored on a storage device at one or more locations. Thus, for example, an index database can contain a number of collections of data, each of which may be organized and accessed differently.

[0123] Similarly, the term "engine" as used herein is used to broadly refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or software components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines can be installed and executed on the same computer, or on the same set of computers.

[0124] The processes and logical flows described herein can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating input data and generating output. Also, the processes and logical flows can be performed by dedicated logic circuitry, such as an FPGA or ASIC, or by a combination of dedicated logic circuitry and one or more programmed computers.

[0125] A computer suitable for executing a computer program can be based on a general-purpose microprocessor or a dedicated microprocessor, or both, or any other kind of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory or a random access memory, or both. Indispensable elements of a computer are a central processing unit for executing or for the execution of instructions, and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, dedicated logic circuitry. Generally, a computer also includes, or is operatively coupled to receive data from, or transfer data to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer need not have such devices. Further, a computer can be embedded in another device, such as, by way of example only, a cellular phone, a personal digital assistant (PDA), a mobile audio player or a mobile video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive.

[0126] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, non-volatile media, and non-volatile memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD ROM disks and DVD-ROM disks.

[0127] To enable interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) monitor or an LCD (liquid crystal display) monitor, and a keyboard and a pointing device, such as a mouse or a trackball, by which the user can provide input to the computer. Other types of devices can also be used to enable interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input received from the user can be received in any form, including acoustic input, voice input, or tactile input. Further, the computer can interact with the user by sending documents to the devices used by the user and receiving documents from such devices. For example, in response to requests received from a web browser on the user's device, the computer can send web pages to the web browser on the user's device. Also, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0128] A data processing apparatus for implementing a machine learning model can include, for example, a dedicated hardware accelerator unit for processing the normal computationally intensive parts of machine learning training or machine learning production, i.e., the inference workload.

[0129] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0130] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes back-end components, such as, for example, a data server, or in a computing system that includes middleware components, such as, for example, an application server, or in a computing system that includes front-end components, such as, for example, a graphical user interface, a web browser, or an app with which a user may interact with an implementation of the subject matter described in this specification, or in a computing system that includes a combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be connected to each other by any form or medium of digital data communication, such as, for example, a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0131] A computing system can include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The relationship between the client and server arises by virtue of computer programs running on respective computers that have a client-server relationship to each other. In some embodiments, the server transmits data, such as, for example, an HTML page, to a user device for the purpose of displaying data to a user who interacts with a device that operates as a client, and for the purpose of receiving user input from such a user. Data generated at the user device, such as, for example, the result of a user interaction, can be received at the server from the device.

[0132] This specification includes details of many specific implementations, which should not be construed as limitations on the scope of the invention, nor as limitations on the possible scope of the claimed subject matter, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Also, some features described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented separately in multiple embodiments, or in any suitable sub - combination. Further, although features may be described above as acting in a certain combination and may even initially be claimed as such, one or more features from the claimed combination may, in some instances, be removed from that combination, and the claimed combination may be directed to a sub - combination, or a variation of a sub - combination.

[0133] Similarly, operations are depicted in the drawings in a particular order and recited in the claims, but this should not be understood as requiring that such operations be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations be performed, for achieving the desired results. In some situations, multitasking and parallel processing may be advantageous. Further, the separation of various system modules and system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the program components and systems described are generally understood to be capable of being integrated together as a single software product, or packaged as multiple software products.

[0134] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Description of Signs

[0135] 100 Reinforcement learning system 102 Agent 104 Environment 106 Action 108 Observation 110 Control neural network system 118 Network parameter 120 Action selection neural network 122 Action selection output 124 Encoder subnet network 126 Encoded representation 128 Attention subnet network 130 Attention subnet network output 132 Gating subnet network 134 Gating subnet network output 136 Recurrent subnet network 140 Action selection subnet network 150 Training engine 402A, 402B Time step 412A, 412B Encoded representation of observation 414A, 414B Attention subnet network training output 410 RL loss 420 Contrast loss

Claims

1. A system for performing a task, comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network configured to perform the task, wherein the neural network comprises: an attention sub-network configured to, at each time step of a plurality of time steps, receive an attention sub-network input generated from an input received by the neural network for the time step, and generate an attention sub-network output based at least on applying an attention mechanism to the attention sub-network input; a recurrent sub-network configured to, at each time step of the plurality of time steps, receive a recurrent sub-network input generated from the attention sub-network output to update a current hidden state of the recurrent sub-network corresponding to the time step, and generate a recurrent sub-network output, wherein the current hidden state is generated by processing one or more previous recurrent sub-network inputs, and the update of the current hidden state changes the current hidden state by processing the received recurrent sub-network input; and an output sub-network configured to, at each time step of the plurality of time steps, receive an output sub-network input generated from the recurrent sub-network output, and process the output sub-network input to generate an output for the task. A system as claimed in claim 1.

2. The neural network of claim 1 further comprises: an encoder sub-network configured to, at each time step of a plurality of time steps, process the input received by the neural network for the time step and generate an encoded representation of the input.

3. The system of claim 2, wherein the attention sub-network input comprises the encoded representation of the input.

4. The system according to any one of claims 1 to 3, wherein the attention mechanism is a masked attention mechanism.

5. The system according to any one of claims 1 to 4, wherein the recurrent subnetwork comprises one or more long short-term memory (LSTM) layers.

6. The output for the task includes respective numerical probability values for each output item in a set of possible output items, wherein performing the task includes selecting an output item for the time step from the set of possible output items based on the respective numerical probability values. The system according to any one of claims 1 to 5.

7. The system according to any one of claims 2 to 6, wherein the neural network further comprises a gating layer configured to apply a gating mechanism to i) the encoded representation of the input and ii) the attention subnetwork output to generate the recurrent subnetwork input.

8. The system according to claim 7, wherein applying the gating mechanism to i) the encoded representation of the input and ii) the attention subnetwork output includes applying a gated recurrent unit (GRU) to i) the encoded representation of the input and ii) the attention subnetwork output.

9. The system according to any one of claims 1 to 8, wherein at each time step of the plurality of time steps, the attention subnetwork input includes the encoded representation of the input and the encoded representations of one or more previous inputs received by the neural network for one or more previous time steps.

10. A method executed by one or more computers, comprising: processing, by an attention subnetwork of a neural network configured to perform a task, an attention subnetwork input generated from an input received by the neural network to generate an attention subnetwork output based at least on applying an attention mechanism to the attention subnetwork input; A step of processing a recurrent sub-network input generated from the attention sub-network output to update the current hidden state of the recurrent sub-network by the recurrent sub-network of the neural network, and generating a recurrent sub-network output, wherein the current hidden state is generated by processing one or more previous recurrent sub-network inputs, and the update of the current hidden state changes the current hidden state by processing the received recurrent sub-network input. A step of processing an output sub-network input generated from the recurrent sub-network output by the output sub-network of the neural network to generate an output for the task A method comprising: **Claim 11** The encoder sub-network of the neural network The method according to claim 10, further comprising a step of processing the input received by the neural network to generate an encoded representation of the input. **Claim 12** The method according to claim 11, wherein the attention sub-network input includes the encoded representation of the input. **Claim 13** The method according to claim 10, wherein the attention mechanism is a masked attention mechanism. **Claim 14** The method according to claim 10, wherein the recurrent sub-network comprises one or more long short-term memory (LSTM) layers. **Claim 15** The output for the task includes respective numerical probability values for each output item in a set of possible output items. Performing the task includes selecting an output item for a time step from the set of possible output items based on the respective numerical probability values. The method according to claim 10. **Claim 16** The method according to claim 11, further comprising a step of applying a gating mechanism to i) the encoded representation of the input and ii) the output of the attention sub-network to generate the recurrent sub-network input by the gating layer of the neural network. **Claim 17**: The method of claim 16, wherein applying the gating mechanism to i) the encoded representation of the input and ii) the attention subnetwork output comprises applying a gated recurrent unit (GRU) to i) the encoded representation of the input and ii) the attention subnetwork output. **Claim 18**: The method of claim 11, wherein the attention subnetwork input comprises the encoded representation of the input and the encoded representation of one or more previous inputs received by the neural network. **Claim 19**: One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform respective operations of any one of the methods of claims 10 to 18.

Images