Neural networks with gated attention

Abstract

System comprising one or more computers and one or more storage devices that store instructions which, when executed by the one or more computers, cause the one or more computers to implement a neural attention network configured to receive network input and process the network input to produce output, wherein the neural attention network comprises: an attention block configured to receive a query input, a key input, and a value input derived from an attention block input, wherein the attention block comprises: a first layer normalization layer that is configured to apply a layer normalization operation to the query input, the key input, and the value input to produce a normalized query input, a normalized key input, and a normalized value input; a neural attention network layer configured to: Receiving an attention layer input that is the normalized query input, The normalized key input and the normalized value input are included, and Applying an attentional mechanism to the attentional layer input to generate an attentional layer output for the neural to create an attention network layer, whereby the neural attention network was trained on a reinforcement learning target.

Description

GENERAL STATE OF THE ART

[0001] This specification refers to a system that processes an input sequence to generate an output using a neural attention network.

[0002] Neural networks are machine learning models that use one or more layers of nonlinear units to predict an output for a given input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input for the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a given input according to the current values ​​of its respective set of parameters. SUMMARY

[0003] This specification describes a neural network system implemented as computer programs on one or more computers at one or more locations, which includes a neural attention network configured to receive an input sequence and process the input sequence to produce an output.

[0004] The neural attention network can include an attention block configured to receive a query input, a key input, and a value input derived from an attention block input. The attention block can include a neural attention network layer.

[0005] The neural attention network layer can be configured to receive an attention layer input derived from the query input, key input, and value input, and to apply an attention mechanism to the attention layer input to generate an attention layer output for the neural attention network layer.

[0006] The attention block can further include a neural gating network layer configured to apply a gating mechanism to the attention block input and the attention layer output of the neural attention network layer to produce a gated attention output. The attention block input can, for example, consist of embeddings from the output of a previous attention block in the neural attention network, or of embeddings derived from the input sequence, or of the input sequence itself.

[0007] The attention block can further include a first layer normalization layer configured to apply a layer normalization operation to the query input, key input, and value input to produce a normalized query input, a normalized key input, and a normalized value input. The attention layer input can include the normalized query input, the normalized key input, and the normalized value input.

[0008] Applying the gating mechanism to the attentional block input and the attentional layer output may involve one or more of the following: applying sigmoid modulation to the attentional block input to produce a first sigmoid-modulated output, and combining the first sigmoid-modulated output with the attentional layer output to produce the gated attentional output; and / or applying sigmoid modulation to the attentional layer output to produce a second sigmoid-modulated output, and combining the second sigmoid-modulated output with the attentional block input to produce the gated attentional output; and / or computing a combination of the attentional block input and the attentional layer output using sigmoid weighting to produce the gated attentional output.and / or applying sigmoid and tanh activation to the attentional layer output to produce a sigmoid-tanh output, and combining the sigmoid-tanh output with the attentional block input to produce the gated attentional output; and / or applying a gated recurrent unit to the attentional block input and the attentional layer output. It should be noted that applying sigmoid modulation is equivalent to applying a sigmoid activation function. It is further understood that combining the attentional block input and the attentional layer output using sigmoid weighting to produce the gated attentional output may be a convex combination.

[0009] The attention block may further include a second layer normalization layer configured to apply a layer normalization operation to the gated attention output to produce a normalized gated attention output. The attention block may further include one or more neural feedforward network layers configured to apply one or more transformations to the normalized gated attention output to produce a temporary attention block output. The attention block may further include a second neural gating network layer configured to apply a second gating mechanism to the temporary attention block output and the gated attention output to produce a final attention block output for the attention block.

[0010] The attentional mechanism can be a self-attentional mechanism. The attentional mechanism can be a masked self-attentional mechanism. A masked self-attentional mechanism is an attentional mechanism that is masked in such a way that it does not consider or otherwise process data that is not located at a position prior to the current position in the attentional layer input sequence. That is, the masked self-attentional mechanism considers or processes data at a position that precedes the current position in the attentional layer sequence.

[0011] The input sequence can be a training input sequence. The neural attention network can process the training input sequence to generate an output. This output can be used as part of a goal function to train the neural attention network. The training input sequence and the goal function can be selected according to the training task. Furthermore, the system can be configured to train the neural attention network.

[0012] According to another aspect, one or more computer storage media are provided that store instructions which, when executed by one or more computers, cause the one or more computers to implement the neural attention network described above.

[0013] According to another aspect, a procedure is provided that includes the operations for which the neural attention network described above is configured.

[0014] It is understood that features described in the context of one aspect can be combined with features described in the context of another aspect.

[0015] Certain embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages. By replacing a residual connection in a neural attention network with a gating function, the techniques described here enable significantly more stable training of the neural attention network and improve learning speed. Training the neural attention network can therefore require fewer computational resources, such as reduced processor cycles, reduced clock cycle time, and reduced power consumption, thus improving the computational efficiency of the training process. Furthermore, the overall performance of the network can also be improved. The final performance of the network is also robust against hyperparameter selections and variations caused by different random seeds.These techniques enable the neural attention network to achieve good results in domains, such as reinforcement learning, where the conventional neural attention network could not. For example, conventional neural attention networks used in reinforcement learning have, in some cases, only achieved performance comparable to a random directive. Furthermore, these techniques can modify how a layer normalization operation is applied within an attention block, allowing the attention block to be initialized with an identity operation at the start of training.This modification can be particularly advantageous in a machine reinforcement learning environment, as it allows a robot agent to begin controlling through a purely reactive policy and, as the learning process progresses, to learn to use longer-term information, thus further accelerating the learning rate. That is, the agent can learn reactive behaviors first before learning memory-based behaviors.

[0016] These techniques are also more scalable and allow the learning of larger and / or deeper networks to handle more complex problems and environments.

[0017] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying figures and the following description. Further features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 shows an exemplary neural network system that includes a neural attention network containing an attention block. Fig. Figure 2 illustrates how a conventional attention block can be modified to include an attention block of the type described in this specification. Fig. Figure 3 is a flowchart of an exemplary process for generating an attention block output for an attention block from an attention block input.

[0018] Identical reference symbols and designations in the different drawings indicate identical elements. DETAILED DESCRIPTION

[0019] This specification describes a neural network system implemented as computer programs on one or more computers at one or more locations, comprising a neural attention network containing one or more attention blocks. The neural network system is configured to receive an input sequence and process the input sequence to produce an output.

[0020] For example, the neural network system can be a reinforcement learning system that selects actions to be performed by a reinforcement learning agent interacting with an environment. For the agent to interact with the environment, the system can receive an input sequence containing a series of observations characterizing different states of the environment. The system can then produce an output specifying one or more actions to be performed by the agent in response to the received input sequence—that is, in response to the last observation in the sequence. In other words, the sequence of observations includes a current observation characterizing the current state of the environment and one or more historical observations characterizing past states of the environment.

[0021] In some implementations, the environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment. For example, the agent might be a robot that interacts with the environment to perform a specific task, such as locating an object of interest in the environment, moving an object of interest to a specific location in the environment, or navigating to a specific destination in the environment; or the agent might be an autonomous or semi-autonomous land, air, or sea vehicle that navigates through the environment.

[0022] In these implementations, observations can include, for example, one or more images, object position data, and sensor data to capture observations while the agent interacts with the environment, such as sensor data from an image, distance, or position sensor, or from an actuator.

[0023] For example, in the case of a robot, observations may include data that characterize the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, for example, gravity-compensated torque feedback, and global or relative pose of an object held by the robot.

[0024] In the case of a robot or other mechanical agent or vehicle, observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. Observations may be defined in one, two, or three dimensions and may be absolute and / or relative.

[0025] The observations may also include, for example, sampled electronic signals such as motor current or a temperature signal; and / or image or video data, for example from a camera or a LIDAR sensor, e.g. data from sensors of the agent or data from sensors that are located separately from the agent in the environment.

[0026] In the case of an electronic agent, the observations may include data from one or more sensors monitoring part of a plant or service facility, such as current, voltage, power, temperature and other sensors and / or electronic signals representing the functioning of electronic and / or mechanical equipment.

[0027] In these implementations, the actions can be control inputs to control the robot, e.g., torques to the robot's joints or higher-level control commands, or the autonomous or semi-autonomous land, air, or sea vehicle, e.g., torques to the control surface or other controls of the vehicle, or higher-level control commands.

[0028] In other words, actions can include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or parts of another mechanical agent. Action data can additionally or alternatively include electronic control data, such as motor control data or more generally, data for controlling one or more electronic devices within the environment, the control of which has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, actions can include actions for controlling navigation, such as steering and movement, e.g., braking and / or accelerating the vehicle.

[0029] In some implementations, the environment is a simulated environment, and the agent is implemented as one or more computers that interact with the simulated environment. Training an agent in a simulated environment can allow the agent to learn from large amounts of simulated training data while avoiding risks associated with training the agent in a real-world environment, such as damaging the agent by performing poorly chosen actions. An agent trained in a simulated environment can then be deployed in a real-world environment.

[0030] For example, the simulated environment can be a simulation of a robot or vehicle, and the reinforcement learning system can be trained on the simulation. For instance, the simulated environment can be a motion simulation environment, such as a driving or flight simulation, and the agent is a simulated vehicle navigating through the motion simulation. In these implementations, the actions can be control inputs to steer the simulated user or vehicle.

[0031] In another example, the simulated environment could be a video game and the agent could be a simulated user playing the video game.

[0032] In another example, the environment can be a chemical synthesis or protein folding environment, such that each state is a respective state of a protein chain or one or more intermediates or precursor chemicals, and the agent is a computer system for determining how the protein chain should fold or how the chemical should be synthesized. In this example, the actions are possible folding actions for folding the protein chain or actions for assembling precursor chemicals / intermediates, and the desired outcome can include, for example, folding the protein so that it is stable and achieves a specific biological function, or providing a valid synthesis route for the chemical. As a further example, the agent can be a mechanical agent that automatically performs or controls the protein folding actions selected by the system without human interaction.The observations may include direct or indirect observations of a protein's state and / or may be derived from a simulation.

[0033] Similarly, the environment can be a drug design environment, such that each state represents a potential pharmacological drug, and the agent is a computer system for determining drug components and / or a synthetic pathway for the drug. The drug / synthesis can be designed based on a reward derived from a drug goal, for example, in a simulation. Alternatively, the agent can be a mechanical agent that performs or controls the synthesis of the drug.

[0034] In some applications, the agent can be a static or mobile software agent, meaning a computer program configured to work autonomously and / or with other software agents or people to perform a task. For example, the environment might be an integrated circuit routing environment, and the system might be configured to learn to perform a routing task to route interconnects of an integrated circuit such as an ASIC. The rewards (or costs) might then depend on one or more routing metrics, such as interconnect resistance, capacitance, impedance, loss, velocity, or propagation delay; physical conductor parameters, such as width, thickness, or geometry; and design rules. The observations might be observations of component positions and connections; the actions might be component placement actions, such as...to define a component position or orientation, and / or to perform connection routing actions, such as connection selection and / or placement actions. The routing task can thus include component placement, i.e., determining the positions and / or orientations of components within the integrated circuit, and / or determining the routing of connections between the components. Once the routing task is complete, an integrated circuit, such as an ASIC, can be manufactured according to the determined placement and / or routing. Alternatively, the environment can be a data packet communication network environment, and the agent can be a router to route data packets across the communication network based on network observations.

[0035] In general, in the case of a simulated environment, the observations may include simulated versions of one or more of the previously described observations or types of observations, and the actions may include simulated versions of one or more of the previously described actions or types of actions.

[0036] In some other applications, the agent can control actions in a real-world environment that includes equipment, such as a data center, a power or water distribution system, a manufacturing plant, or a service facility. The observations can then relate to the operation of the plant or facility. For example, the observations might include observations of power or water usage by equipment, observations of power generation or distribution control, or observations of resource utilization or waste production. The agent can control actions in the environment to increase efficiency, such as by reducing resource use, and / or to reduce the environmental impact of operations in the environment, such as by reducing waste.The actions may include actions that control or impose operating conditions for equipment of the plant / facility, and / or actions that result in changes to settings in the operation of the plant / facility, e.g. to adjust or switch components of the plant / facility on / off.

[0037] In some other applications, the environment is a real-world setting, and the agent manages the distribution of tasks across computing resources, such as on a mobile device and / or in a data center. In these implementations, the actions may include assigning tasks to specific computing resources.

[0038] As another example, actions can include presenting advertisements, observations can include ad impressions or a click-through count or rate, and rewards can characterize previous selections of articles or content made by one or more users.

[0039] In general, in the applications described above, where the environment is a simulated version of a real-world environment, once the system / method has been trained in the simulation, it can then be applied to the real environment. That is, control signals generated by the system / method can be used to control the agent to perform a task in the real environment in response to observations from the real environment. Optionally, the system / method can continue training in the real environment based on one or more rewards from the real environment.

[0040] Optionally, in any of the above implementations, the observation at any given time step can include data from a previous time step that may be useful in characterizing the environment, e.g., the action performed at the previous time step, the reward received at the previous time step, and so on.

[0041] In another example, the neural network system can be a neural machine translation system. That is, if the input sequence is a sequence of words in an original language, e.g., a sentence or phrase, the output can be a translation of the input sequence into a target language, i.e., a sequence of words in the target language that represents the sequence of words in the original language.

[0042] As another example, a neural network system can be a speech recognition system. That is, if the input sequence is a sequence of audio data representing a spoken utterance, the output can be a sequence of graphemes, characters, or words representing the utterance—that is, a transcription of the input sequence.

[0043] As another example, the system could be a natural language processing system. For instance, if the input sequence is a sequence of words in an original language, such as a sentence or phrase, the output could be a summary of the input sequence in the original language—that is, a sequence with fewer words than the input sequence but retaining the essential meaning of the input sequence. As another example, if the input sequence is a sequence of words forming a question, the output could be a sequence of words forming an answer to the question. As yet another example, the task could be a natural language comprehension task, such as…an entailment task, a paraphrase task, a text similarity task, a mood task, a sentence completion task, a grammaticity task, and so on, which works on a text sequence in a natural language to produce an output that predicts a property of the text.

[0044] As another example, the system can be part of a computerized medical diagnostic system. For instance, the input sequence could be a sequence of data from an electronic health record, and the output could be a sequence of predicted treatments.

[0045] As another example, the system can be part of an image processing system. For instance, the input sequence can be an image, i.e., a sequence of color values ​​from the image, and the output can be a text sequence describing the image. Alternatively, the input sequence can be a text sequence or other context, and the output can be an image describing that context.

[0046] Fig. Figure 1 shows an exemplary neural network system 100. The neural network system 100 is an example of a system that is implemented as computer programs on one or more computers at one or more locations, in which the systems, components and techniques described below can be implemented.

[0047] Neural network system 100 receives an input sequence 102 and processes the input sequence 102 to generate an output 122. Neural network system 100 includes a neural attention network 110. Neural attention network 110 includes an attention block 124.

[0048] While Fig. As Figure 1 illustrates an attention block 124, the neural attention network 110 can include multiple attention blocks arranged sequentially in a stack and, optionally, other components. Specific examples of neural attention network architectures that include multiple attention blocks and can be modified to include attention blocks of the type described in this specification (e.g., the type of attention block 124) are described in Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, 2019; Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653 / v1 / P19-1285. https: / / www.aclweb.org / anthology / P19-1285; and Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 5998-6008, 2017. URL https: / / papers.nips.cc / paper / 7181-attention-is-all-you-need.pdf.

[0049] Fig. Figure 2 illustrates how a conventional attention block can be modified to include an attention block of the type described in this specification.

[0050] As in Fig. As shown in Figure 2(a), a conventional attention block 200 (also referred to as a “transformer block”) within a conventional neural attention network (or a “neural transformer network”) generally comprises two submodules: a neural attention network layer (e.g., a neural multi-head attention network layer 202), followed by a neural feedforward network (e.g., a positional multilayer perceptron network 208). The input for the transformer block 200 is an embedding(s) from the preceding block E. (l-1) ∈ ℝ T×Dof the neural attention network, where T is the number of time steps, D is the hidden dimension, and L ∈ [0, L] is the block index, where L is the total number of transformer blocks of the neural attention network. E(0) can be assumed to be an input embedding of dimension [T,D], e.g., a word embedding in the case of language modeling or an embedding of observations per time step in a reinforcement learning environment.

[0051] The exemplary transformer block 200 in Fig. 2 includes a neural multi-head attention network layer (MHA network layer) 202, which performs h soft attention operations in parallel on an input E (l-1) calculated for each time step: Y¯(l)=MultiHeadAttention(E(l−1)).

[0052] In particular, to implement multi-head attention, layer 202h employs different attentional mechanisms in parallel. In other words, the neural MHA network layer 202h includes different attentional sublayers, each receiving the same initial queries Q, initial keys K, and initial values ​​V. These initial queries Q, initial keys K, and initial values ​​K are derived from the input of the transformer block.

[0053] Each attentional sublayer is configured to transform the original queries, keys, and values ​​using learned linear transformations, and then to apply an attentional mechanism to the transformed queries, keys, and values. Each attentional sublayer will generally learn different transformations than every other attentional sublayer.

[0054] Specifically, each attention sublayer is configured to apply a learned linear query transformation to each original query to generate a layer-specific query for each original query; to apply a learned linear key transformation to each original key to generate a layer-specific key for each original key; and to apply a learned linear value transformation to each original value to generate a layer-specific value for each original value. The attention sublayer then applies the attention mechanism using these layer-specific queries, keys, and values ​​to generate initial outputs for the neural MHA network layer 202.

[0055] The neural MHA network layer 202 then combines the initial outputs of the attention sublayers to produce the final output of layer 202. For example, layer 202 concatenates the initial outputs of the attention sublayers and applies a learned linear transformation to the concatenated output to generate the output of layer 202.

[0056] Multi-head attention is described in more detail in Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Lion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 5998–6008, 2017.

[0057] After the output of the neural MHA network layer 202 is generated, a residual connection 204 and layer normalization 206 are then applied to the output Y. (l) the neural multi-head attention network layer 202 is applied as follows: Y^(l)=E(l−1)+Y¯(l).Y(l)=LayerNorm(Y^(l)).

[0058] The transformer block 200 further includes a positional multilayer perceptron network (MLP network) 208, which is a temporal 1 x 1 convolution network f (l) (i.e., kernel size 1, step size 1) is applied over each step in the sequence, generating a new embedding tensor E (l) ∈ ℝ T×D : E¯(l)=f(l)(Y(l)).

[0059] After the MLP network 208, there is an update via a residual connection 210, followed by an update via a layer normalization layer 212: E(l)=Y(l)−E¯(l).E(l)=LayerNorm(E(l)).

[0060] To enable much more stable training of a neural attention network and to improve learning speeds, the new type of attention blocks described in this specification features a gating mechanism (or gating function) that replaces a residual connection in conventional transformer blocks. For example, Fig. 2(b) a new attention block 250, which is based on the transformer block 200 and has been modified according to the techniques described in this specification. In Fig. 2(b) A neural gating network layer 218 and a neural gating network layer 224 replace a residual connection 204 and a residual connection 210, respectively, in the transformer block 200. This technique enables the neural attention network to achieve good results in domains, such as reinforcement learning, where the conventional neural attention network could not. Additionally, in the type of attention blocks described in this specification, the layer normalization layers in conventional transformer blocks are rearranged such that layer normalization is applied only to the input stream of each of the submodules. For example, as in Fig. As shown in Figure 2(b), layer normalization layer 214 is applied to the input stream of neural multi-head attention network layer 216, and layer normalization layer 220 is applied to the input stream of positional MLP network 222. This differs from layer normalization layers 206 and 212 being applied to the output stream of neural multi-head attention network layer 202 and the output stream of positional MLP network 208 of transformer block 200, respectively, in Figure 2(b). Fig. 2(a) can be applied. A key advantage of this rearrangement is that it now allows an identity mapping from the transformer input at the first layer to the transformer output after the last layer. This contrasts with the conventional transformer, where there are a number of layer normalization operations that nonlinearly transform the state encoding. This modification can be particularly advantageous in a reinforcement learning environment, as it allows a robot agent to begin controlling by a purely reactive policy and, as it learns, to use longer-term information, thus providing a further acceleration of the learning rate.Since the reordering of layer normalization causes a path where two linear layers are applied sequentially, in the new type of attention blocks described herein, a Rectified Linear Unit (ReLU) activation is applied to each submodule output prior to the residual connection.

[0061] The techniques described above enable a neural attention network exhibiting the novel type of attention blocks described herein to learn much faster and more reliably, and to demonstrate significantly better final performance than canonical neural transformer networks.

[0062] It is understood that the new type of attention block can be constructed from a combination of one or more of the transformations described above and may not require all of them. The network architecture and operations of the new type of attention block (e.g., attention block 124) are described below with reference to Fig. 1 described in detail.

[0063] As in Fig. As shown in Figure 1, attention block 124 includes a neural attention network layer 108 and a neural gating network layer 112. Attention block 124 is configured to receive an attention block input 104. The attention block input 104 can be embeddings from the output of a previous attention block in neural attention network 110 or embeddings derived from the input sequence 102. For example, neural attention network 110 includes a neural encoder network (e.g., an MLP network or a convolutional neural network) that generates embeddings (e.g., word embeddings in the case of language modeling or observation-per-time-step embeddings in a reinforcement learning environment) from the input sequence 102, which are used as input by the first attention block of neural attention network 110.As another example, the neural attention network 110 can generate the embeddings as input to the first attention block in a different way, e.g., using a lookup table. The neural attention network layer 108 is configured to receive queries, keys, and values ​​derived from the attention block input 104 as attention layer input.

[0064] The queries, keys, and values ​​can differ for different types of attention. That is, different types of neural attention network layers use different sources for the initial queries, keys, and values ​​that are received as input by neural attention network layer 108.

[0065] In particular, if the neural attention network layer 108 is an encoder self-attention layer in an encoder that has multiple subnetworks, all keys, values, and queries originate from the same location—in this case, the output of the previous subnetwork in the encoder, or, for the encoder self-attention layer in the first subnetwork, the embeddings of the inputs—and each position in the encoder can consider all positions in the input sequence. Thus, for each position in the input sequence, there is a corresponding key, value, and query.

[0066] If neural attention network layer 108 is a decoder self-attention layer in a decoder with multiple subnetworks, each position in the decoder considers all positions in the decoder that precede it. Thus, all keys, values, and queries originate from the same source—in this case, the output of the previous subnetwork in the decoder, or, for the decoder self-attention layer in the first decoder subnetwork, the embeddings of previously generated outputs. Therefore, there is a corresponding key, value, and query for each position in the output order preceding the current position.

[0067] If neural attention network layer 108 is an encoder-decoder attention layer, the queries originate from the preceding component in the decoder, and the keys and values ​​come from the encoder's output—that is, from the coded representations generated by the encoder. This allows each position in the decoder to consider all positions in the input sequence. Thus, there is a corresponding query for each position in the output sequence preceding the current position, and a corresponding key and value for each position in the input sequence.

[0068] In some implementations, in order to initialize attention block 124 with an identity operation, attention block 124 may include a first layer normalization layer 106 configured to apply a layer normalization operation to the query input, key input, and value input to produce a normalized query input, a normalized key input, and a normalized value input. In these implementations, neural attention network layer 108 receives the normalized query input, the normalized key input, and the normalized value input as the attention layer input.

[0069] Layer normalization is generally used to improve training speeds for various neural network models. Unlike batch normalization, which uses the distribution of a summed input to a neuron across a minibatch of training cases to calculate a mean and variance that are then used to normalize the summed input to the neuron in each training case, layer normalization calculates the mean and variance used for normalization from all the summed inputs to the neurons in a layer in a single training case. Layer normalization is described in detail in Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv: 1607.06450, 2016.

[0070] Neural attention network layer 108 is then configured to apply an attention mechanism to the attention layer input in order to generate an attention layer output for neural attention network layer 108. In some implementations, the attention mechanism is a self-attention mechanism. In some other implementations, the attention mechanism is a masked self-attention mechanism. A masked self-attention mechanism is an attention mechanism that is masked in such a way that it does not consider or otherwise process data that is not located at a position prior to the current position in the attention layer input sequence.

[0071] For example, the attention layer output, denoted as Y (l) , from attention block input 104, designated as E (l-1) , calculated as follows: Y¯(l)=MultiHeadAttention(LayerNorm([StopGrad(M(l−1),E(l−1)])), where M (l-1) ∈ R LxD`<>` is an additional memory tensor that is treated as constant during weight updates; `<>` is a stop gradient function that prevents gradients from flowing backward during backpropagation; and `<>` is a basic multi-head attention mechanism, or multi-head attention, that uses relative positional codings. Unlike a basic MHA operation, which does not explicitly consider sequence order because it is permutation-invariant, positional codings do consider sequence order and can be used in domains such as language, where sequence order is an important semantic cue. Examples of positional codings are described in Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. (Transformer-XL: Attentive language models beyond a fixed-length context)In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653 / v1 / P19-1285. URL https: / / www.aclweb.org / anthology / P19-1285.

[0072] Neural gating network layer 112 is configured to apply a gating mechanism to the attentional block input and attentional layer output of neural attention network layer 108 to produce a gated attentional output. For example, neural gating network layer 112 can implement a gating mechanism gMHA(l) on the attention block input E (l-1) and the attention layer output Y (l) Apply to the neural attention network layer 108 to generate a gated attention output Y (l) to be generated as follows: Y(l)=gMHA(l)(E(l−1)⋅ReLU(Y¯(l))). where ReLU() is a ReLU activation applied to the attention layer output Y (l) Examples of gating mechanisms are described in detail below.

[0073] In some implementations, the attention block 124 further includes a second-layer normalization layer 114, which is configured to apply a second-layer normalization operation to the gated attention output of the neural gating network layer 112 to produce a normalized gated attention output. In these implementations, the attention block 124 may include one or more neural feedforward network layers 116 (e.g., a positional multilayer network) configured to apply one or more transformations to the normalized gated attention output to produce a temporary attention block output. For example, the temporary attention block output may be denoted as E (l) and are calculated as follows: E¯(l)=f(l)(LayerNorm(Y(l))), where LayerNorm() is the second layer normalization operation applied to the gated attention output Y (l) , and f (l) () denotes one or more transformations that affect the normalized gated attention output LayerNorm (Y (l) ) are applied to generate the temporary attention block output E (l) For example, one or more transformations may involve applying a temporal 1 x 1 convolutional network (i.e., kernel size 1, step size 1) to the normalized gated attention output.

[0074] Attention block 124 can include a second neural gating network layer 118 configured to apply a second gating mechanism to the temporary attention block output and the gated attention output to generate a final attention block output 120 for attention block 124. For example, the second neural gating network layer 118 can implement a second gating mechanism. gMLP(l) on the temporary attention block issue E (l) and the gated attention output Y (l) Apply the neural gating network layer 112 to obtain the final attention block output E (l) to be generated as follows: E(l)=gMLP(l)(Y(l)⋅ReLU(E¯(l))). where ReLU() is a ReLU activation applied to the attention block output E (l) .

[0075] It is noted that the residual compound in equations 2 and 4, which uses the conventional transformer block of Fig. 2 is associated with being replaced by the neural gating network layer 112 and the second neural gating network layer 118 to improve the performance and optimization stability of the resulting neural attention network.

[0076] There are a variety of gating mechanisms that could be applied by the neural gating network layer 112 and the second neural gating network layer 118. In general, a gating mechanism computes a learned combination of inputs for the neural gating network layer to produce a gated output for the neural gating network layer.

[0077] For example, in some implementations, the neural gating network layer 112 and / or the second neural gating network layer 118 can apply a sigmoid modulation σ () to the respective input stream x (e.g., the attentional block input and / or the gated attentional output) to produce a first sigmoid-modulated output, and combine the first sigmoid-modulated output with the respective output stream y (e.g., the attentional layer output and / or the temporary attentional block output) to produce a gated output g. (l) (x, y) (e.g., the gated attention output and / or the final attention block output) to be generated as follows: g(l)(x,y)=σ(Wg(l)x)⊙x+y.

[0078] In some other implementations, the neural gating network layer 112 and / or the second neural gating network layer 118 can apply a sigmoid modulation σ() to the output stream y to produce a second sigmoid-modulated output, and combine the second sigmoid-modulated output with the input stream x to produce a gated output as follows: g(l)(x,y)=x−σ(Wg(l)x−bg(l))⊙y.

[0079] In some other implementations, the neural gating network layer 112 and / or the second neural gating network layer 118 can use a highway connection that modulates both input and output streams with a sigmoid modulation, i.e., by calculating a convex combination of the input and output streams using a sigmoid weighting to produce a gated output: g(l)(x,y)=σ(Wg(l)x+bg(l))⊙x+(1−σ(Wg(l)x−bg(l)))⊙y, where Wg(l) and bg(l) Weights are defined as element-wise multiplication. In some implementations, the initial setting can be... bg(l)>0 significantly improve learning speed.

[0080] In some other implementations, the neural gating network layer 112 and / or the second neural gating network layer 118 can apply a sigmoid and a tanh activation to the output stream γ to produce a sigmoid-tanh output, and combine the sigmoid-tanh output with the input stream to produce a gated output: g(l)(x,y)=x−σ(Wg(l)g−b)⋅tanh(Ug(l)g). where Ug(l) represents a weight.

[0081] In some implementations, neural gating network layer 112 and / or the second neural gating network layer 118 can apply a Gated Recurrent Unit (GRU) to the input and output streams. The GRU is a recurrent network that functions similarly to a Long Short-Term Memory (LSTM) network with a forget gate, but has fewer parameters than LSTM because it lacks an output gate. In some implementations, this gating mechanism can be adjusted in depth as an unbound activation function. This means that, although the GRU is a recurrent neural network (RNN), the gating mechanism can use the same formula that GRUs use to update their hidden states over time to update the skip connections at each neural gating network layer of neural attention network 110 instead.Thus, an update performed by the gating mechanism described here could be viewed as an update of a GRU-RNN that unfolds over the depth of the network 110, rather than unfolding over time. However, instead of having a single set of "bound" or "shared" parameters for this GRU update at each neural gating network layer (which would be the case if a single GRU-RNN were used), there are different sets of parameters for the GRU update at each neural gating network layer.

[0082] There are several variants of the full Gated Recurrent Unit, where the gating operation is performed using a previous hidden state of the recurrent network and a bias in various combinations, as well as a simplified form called the minimal Gated Unit, as described below. Complete Gated Recurrent Unit:

[0083] Initially, for t=0, the output vector h0=0. zt=σg(Wzxt+Uzht−1+bz) rt=σg(Wrxt+Urht−1+br) h^t=ϕh(Whxt+Uh(rt⊙ht−1)+bh) ht=(1−zt)⊙ht−1+zt⊙h^t

[0084] where x t an input vector, h t an output vector, ĥ t a candidate activation vector, z t an update gate vector, z t A reset gate vector, W, U, and b are parameter matrices and vector. σ g is a sigmoid function and ϕ h a hyperbolic tangent. Alternative activation functions are possible, provided that σ g (x) ∈ [0,1]. Alternative forms of the equations (*) can be obtained by changing z t and are created. Minimum Gated Recurrent Unit:

[0085] The minimal gated unit is similar to the full gated unit, except that the update and reset gate vectors are combined into a forget gate. This also implies that the equation for the output vector must be changed. ft=σg(Wfxt+Ufht−1+bf) h^t=ϕh(Whxt+Uh(ft⊙ht−1)+bh) ht=(1−ft)⊙ht−1+ft⊙h^t where x t an input vector is h t an output vector is, ĥ t a candidate activation vector is, f t is a forgetting vector, W, U and b are parameter matrices and vector.

[0086] The final attention block output 120 can be provided as input for the next attention block or other components of the neural attention network 110 for further processing, or can be used to generate the output 122 of the neural network system 100.

[0087] The neural attention network 110 can include one or more output layers configured to receive the output of the final attention block from the one or more attention blocks within the neural attention network 110. The one or more output layers are configured to process the output of the final attention block to generate the output 122 of the neural network system 100.

[0088] Fig. Figure 3 is a flowchart of an exemplary process for generating an attention block output for an attention block of a neural attention network from an attention block input. For simplicity, process 300 is described as being executed by a system of one or more computers located at one or more sites. For example, a neural network system, e.g., neural network system 100, may be... Fig. 1, which is appropriately programmed according to this specification, perform process 300.

[0089] The system receives a query input, a key input, and a value input derived from an attention block input (step 302). In some implementations, the system can derive the attention block input from an output of a previous attention block of the neural attention network. In other implementations, the attention block input can be derived from an input sequence of a neural network system that includes the neural attention network. The derivation of the attention block input can be performed in accordance with techniques described in the references cited above.

[0090] The system receives, through a neural attention network layer of an attention block, an attention layer input derived from the query input, the key input, and the value input (step 304).

[0091] In some implementations, to initialize the attentional block for an identity operation, the system applies a layer normalization operation to the attentional block input, which includes the query input, key input, and value input, using a first layer normalization layer of the attentional block to generate a normalized query input, a normalized key input, and a normalized value input. In these implementations, the neural attentional network layer receives the normalized query input, the normalized key input, and the normalized value input as the attentional layer input.

[0092] Using the neural attention network layer, the system applies an attention mechanism to the attention layer input to generate an attention layer output for the neural attention network layer (step 306). In some implementations, the attention mechanism is a self-attention mechanism. In some other implementations, the attention mechanism is a masked self-attention mechanism.

[0093] Using a neural gating network layer of the attention block, the system applies a gating mechanism to the attention block input and the attention layer output of the neural attention network layer to produce a gated attention output (step 308).

[0094] In some implementations, applying a gating mechanism specifically involves applying sigmoid modulation to the attention layer input to produce an initial sigmoid-modulated output, and combining the initial sigmoid-modulated output with the attention layer output to produce the gated attention output.

[0095] In some other implementations, applying a gating mechanism involves applying sigmoid modulation to the attention layer output to produce a second sigmoid-modulated output, and combining the second sigmoid-modulated output with the attention layer input to produce the gated attention output.

[0096] In some other implementations, applying a gating mechanism involves calculating a convex combination of the attention layer input and the attention layer output using sigmoid weighting to generate the gated attention output.

[0097] In some other implementations, applying a gating mechanism involves applying sigmoid and tanh activation to the attention layer output to produce a sigmoid-tanh output, and combining the sigmoid-tanh output with the attention layer input to produce the gated attention output.

[0098] In some other implementations, applying a gating mechanism involves applying a Gated Recurrent Unit (GRU) to the attention layer input and output. The GRU is a recurrent network that functions similarly to a Long Short-Term Memory (LSTM) network, but with fewer parameters. This gating mechanism can be adjusted in depth as an unbounded activation function.

[0099] In some implementations, the system, using a second layer normalization layer of the attention block, applies a layer normalization operation to the gated attention output to produce a normalized gated attention output (step 310).

[0100] In some implementations, the system, using a neural feedforward network layer of the attention block, applies one or more transformations to the normalized gated attention output to generate a temporary attention block output (step 312). For example, the one or more transformations may involve applying a temporal 1 × 1 convolutional network (i.e., kernel size 1, step size 1) to the normalized gated attention output.

[0101] In some implementations, using a second neural gating network layer of the attention block, the system applies a second gating mechanism to the temporary attention block output and the gated attention output to generate an attention block output for the attention block (step 314).

[0102] The system can provide the attention block output as input for the next attention block or other components of the neural attention network for further processing. If the attention block is the last attention block in the neural attention network, the system can provide the attention block output to one or more output layers of the neural attention network that are configured to map the attention block output to the system's output.

[0103] The neural attention network described in this specification can be trained on training data using conventional machine learning techniques (e.g., backpropagation) to optimize a target function suitable for the task for which the neural attention network is configured.By replacing a residual connection in conventional attention blocks with a gating function and by modifying the way a layer normalization operation is applied within an attention block to allow the attention block to be initialized to an identity operation at the start of training, the techniques described in this specification enable the training of the neural attention network to become much more stable and improve learning speeds (compared to training conventional neural attention networks with conventional attention blocks).

[0104] In this description, the term "configured" is used in connection with systems and computer program components. For a system consisting of one or more computers to be configured to perform certain operations or actions, this means that software, firmware, hardware, or a combination thereof is installed on the system which, when operating, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform certain operations or actions, this means that the one or more programs contain instructions which, when executed by a data processing device, cause the device to perform those operations or actions.

[0105] Embodiments of the subject matter and the functional operations described in this description may be implemented in digital electronic circuits or in physically embodied computer software, firmware, or hardware, including the structures disclosed in this description and their structural equivalents, or in combinations of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a physical non-transitory storage medium for execution by, or control of the operation of, a data processing device.The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device for direct or serial access, or a combination of one or more of these. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal, which is produced to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0106] The term "data processing device" refers to data processing hardware and encompasses all types of devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may also consist of or include specialized logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may optionally include code that creates an execution environment for the respective computer program, such as code representing processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0107] A computer program, which may also be called or described as a program, software, software application, app, module, software module, script, or code, may be written in any programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be provided in any form, including as a standalone program or module, component, subroutine, or any other unit suitable for use in a computer environment. A program may correspond to a file in a file system, but it need not. A program may be stored in a portion of a file that contains other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in several coordinated files, such as...Files that store one or more modules, subprograms, or code sections. A computer program can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and connected by a data communication network.

[0108] In this description, the term "database" is used in its broadest sense to refer to any collection of data: The data need not be structured in a particular way, or at all, and can be stored on storage devices in one or more locations. Thus, for example, the index database can contain several data collections, each organized differently and accessible in different ways.

[0109] In this description, the term "machine" is similarly used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, a machine is implemented as one or more software modules or components and installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular machine; in other cases, multiple machines may be installed and run on the same computer or computers.

[0110] The processes and logic flows described in this document can be executed by one or more programmable computers running one or more computer programs to perform functions by processing input data and generating outputs. The processes and logic flows can also be executed by dedicated logic circuitry, such as an FPGA or an ASIC, or by a combination of dedicated logic circuitry and one or more programmed computers.

[0111] Computers suitable for running a computer program may be based on microprocessors of a general or special type, or on both, or on any other type of central processing unit. Generally, a central processing unit receives instructions and data from read-only memory, random-access memory, or both. The essential elements of a computer are a central processing unit for carrying out or executing instructions and one or more storage devices for storing instructions and data. The central processing unit and memory may be augmented by or incorporated into special logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as hard drives, hard drives, or flash drives.magnetic, magneto-optical, or optical data carriers, or is functionally coupled to one or more of these devices to receive data from them, transmit data to them, or both. However, a computer does not necessarily have to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a portable audio or video player, a game console, a GPS (Global Positioning System) receiver, or a portable storage device, such as a USB (Universal Serial Bus) flash drive, to name just a few.

[0112] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks and CD-ROM and DVD-ROM disks.

[0113] To provide interaction with a user, the embodiments of the item described in this description can be implemented on a computer that has a display device, e.g., a monitor with a CRT (cathode ray tube) or LCD (liquid crystal display), for showing information to the user, as well as a keyboard and a pointing device, e.g., a mouse or a trackball, with which the user can provide input to the computer. Other types of devices can also be used to interact with a user; for example, the feedback provided to the user can be any form of sensory feedback, e.g., visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, verbal, or tactile input.Furthermore, a computer can interact with a user by sending and receiving documents to and from a device used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received by the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving reply messages from the user.

[0114] Data processing devices for implementing machine learning models may, for example, also include special hardware accelerator units for processing general and computationally intensive parts of machine learning training or production, i.e., inference workloads.

[0115] Machine learning models can be implemented and deployed using a machine learning framework, such as a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.

[0116] Implementations of the object described herein may be found in a computer system that includes a back-end component, such as a data server; a middleware component, such as an application server; a front-end component, such as a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein; or any combination of such back-end, middleware, or front-end components. The system components may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), such as the internet.

[0117] The computing system can include clients and servers. A client and a server are generally located remotely and typically interact via a communication network. The client-server relationship arises from computer programs running on the respective computers, which have a client-server relationship with each other. In some embodiments, a server transmits data, such as an HTML page, to a user device, for example, for the purpose of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, for example, as a result of user interaction, can be received by the device at the server.

[0118] Although this description contains many specific implementation details, these should not be interpreted as limitations on the scope of protection of an invention or on the scope of what can be claimed, but rather as descriptions of features that may be specific to certain embodiments of certain inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in several embodiments or in any suitable subcombination.Furthermore, although features described above may be effective in certain combinations and may even be initially claimed as such, in some cases one or more features from a claimed combination may be extracted from the combination, and the claimed combination may be directed to a subcombination or a variation of a subcombination.

[0119] Similarly, while operations are depicted in the drawings and listed in the claims in a specific order, this should not be interpreted as requiring that such operations be performed in the specific order shown or in sequential order, or that all depicted operations must be performed to achieve desirable results. Multitasking and parallel processing may be advantageous under certain circumstances. Furthermore, the separation of different system modules and components in the embodiments described above should not be interpreted as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0120] Certain embodiments of the object have been described. Other embodiments are within the scope of the following claims. For example, the actions mentioned in the claims can be performed in a different order and still achieve desirable results. As an example, the processes shown in the accompanying figures do not necessarily require the specific order shown or a sequential order to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Zitierte Nicht-Patentliteratur

[0000] Jacob Devlin, Ming-Wei Chang, Kenton Lee und Kristina Toutanova beschrieben. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019

[0048] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653 / v1 / P19-1285. https: / / www.aclweb.org / anthology / P19-1285

[0048] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 5998-6008, 2017. URL https: / / papers.nips.cc / paper / 7181-attention-is-all-you-need.pdf

[0048] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser und Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 5998-6008, 2017

[0056] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le und Ruslan Salakhutdinov beschrieben. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653 / v1 / P19-1285. URL https: / / www.aclweb.org / anthology / P19-1285

[0071]

Claims

[1] System comprising one or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to implement a neural attention network configured to receive network input and process the network input to produce output, wherein the neural attention network comprises: an attention block configured to receive a query input, a key input, and a value input derived from an attention block input, wherein the attention block comprises: a first layer normalization layer that is configured to apply a layer normalization operation to the query input, the key input, and the value input to produce a normalized query input, a normalized key input, and a normalized value input; a neural attention network layer configured to: Receiving an attention layer input that is the normalized query input, The normalized key input and the normalized value input are included, and Applying an attentional mechanism to the attentional layer input to generate an attentional layer output for the neural to create an attention network layer, whereby the neural attention network was trained on a reinforcement learning target. [2] System according to claim 1, wherein the attention block further comprises: an additional neural network layer configured to process the attention block input and attention layer output of the neural attention network layer to generate an intermediate attention output. [3] System according to claim 2, wherein the processing of the attention block input and the attention layer output comprises: Applying sigmoid modulation to the attention block input to produce an initial sigmoid-modulated output; and Combining the first sigmoid-modulated output with the Attention layer output to generate intermediate attention output. [4] System according to claim 2 or 3, wherein the processing of the attention block input and the attention layer output comprises: Applying sigmoid modulation to the attention layer output to generate a second sigmoid-modulated output; and Combining the second sigmoid-modulated output with the attention block input to generate the intermediate attention output. [5] System according to one of claims 2-4, comprising processing the attention block input and the attention layer output: Computation of a convex combination of attention block input and attention layer output using sigmoid weighting to generate the intermediate attention output. [6] System according to any one of claims 2-5, comprising processing the attention block input and the attention layer output: Applying sigmoid and tanh activation to the Attention layer output to generate a sigmoid tanh output, and Combining the sigmoid tanh output with the attention block input to generate the intermediate attention output. [7] System according to any one of claims 2-6, comprising processing the attention block input and the attention layer output: Applying a Gated Recurrent Unit to the attention block input and attention layer output, [8] System according to any one of claims 2-7, wherein the intermediate attention output is a gated attention output, and wherein the attention block further comprises: a second layer normalization layer configured to apply a second layer normalization operation to the gated attention output, to generate a normalized gated attention output, one or more neural feedforward network layers configured to apply one or more transformations to the normalized gated attention output to create a temporary To generate attention block output, and a second neural gating network layer configured to apply a second gating mechanism to the temporary attention block output and the gated attention output to produce a final To generate attention block output for the attention block. [9] System according to any of the preceding claims, wherein the attention mechanism is a self-attention mechanism. [10] System according to any of the preceding claims, wherein the attention mechanism is a masked self-attention mechanism. [11] One or more non-transitory computer storage media that store instructions which, when executed by one or more computers, cause the one or more computers to perform operations to process an attention block input of an attention block of a neural attention network, wherein the operations include: Receiving a query input, a key input, and a value input derived from the attention block input; Applying a layer normalization operation to the query input, the key input, and the value input to produce a normalized query input, a normalized key input, and a normalized value input; Received, through a neural attentional network layer of an attentional block, an attentional layer input comprising the normalized query input, the normalized key input, and the normalized value input; and Applying, using the neural attention network layer, an attention mechanism to the attention layer input to generate an attention layer output for the neural attention network layer, where the neural attention network was trained on a reinforcement learning goal. [12] One or more non-transitory computer storage media according to claim 11, wherein the operations further comprise: Processing, using an additional neural network layer, the attention block input and the attention layer output of the neural attention network layer to generate an intermediate attention output. [13] One or more non-transitory computer storage media according to claim 11 or 12, wherein the intermediate attention output is a gated attention output, and wherein the operations further comprise: Applying a second layer normalization operation to the gated attention output to produce a normalized gated attention output; Applying one or more transformations to the normalized gated attention output to create a temporary attention block output; and Apply, using a second neural gating network layer of the attention block, a second gating mechanism to the temporary attention block output and the gated attention output to generate an attention block output for the attention block.

Images