Attention-based decoder-only sequence conversion neural network

By using self-attention decoder neural network and local or memory compression attention mechanism in neural networks, the problems of low long-sequence conversion efficiency and waste of computing resources in the existing technology are solved, and more efficient and accurate sequence conversion tasks are achieved. .

CN119005252APending Publication Date: 2024-11-22GOOGLE LLC

Patent Information

Application Number
CN202410983002.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2017-10-27
Filing Date
2018-10-29
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing neural networks are incomputed and efficient in processing and generating long sequence conversion tasks, have long training time, and are difficult to effectively learn dependencies between long-distance locations, resulting in poor performance in many sequence conversion tasks.

Method used

The self-attention-based decoder neural network is adopted to directly generate output sequences through the self-attention mechanism, which reduces the dependence on recursive neural networks. Local attention or memory is used to compress attention to reduce calculation costs, and avoids the encoder network. Number of parameters and calculation resource consumption.

Benefits of technology

It realizes more efficient long-sequence conversion, reduces computing resource consumption, improves training and inference speed, can learn more accurately the dependencies between long-distance locations, and improves the performance of sequence conversion tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005252A_ABST
    Figure CN119005252A_ABST
Patent Text Reader

Abstract

The invention relates to an attention-based decoder-only sequence conversion neural network. Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating, for the generation time step, a combined sequence including an input sequence followed by an output token that has been generated up to the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output defining a score distribution with respect to a set of possible output tokens; and selecting an output token from the set of possible output tokens as a next output token in the output sequence using the time step output.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201880075803.X, filed on October 29, 2018.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application claims priority to U.S. patent application serial number 62 / 578,358, filed on October 27, 2017, the entire contents of which are incorporated herein by reference. Technical Field

[0005] This specification relates to transducing sequences using neural networks. Background Art

[0006] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, which is either the next hidden layer or the output layer. Each layer of the network generates an output based on the received input according to the current values ​​of the corresponding set of parameters. Summary of the invention

[0007] This specification describes a system implemented as a computer program on one or more computers at one or more locations that generates an output sequence from an input sequence, the output sequence including a corresponding output at each of a plurality of positions in the output order, the input sequence including a corresponding input at each of a plurality of positions in the input order, i.e., converts the input sequence into the output sequence. In particular, the system generates the output sequence using a self-attention based decoder neural network.

[0008] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0009] The decoder-only architecture of the system described in this specification can efficiently and scalably attend to very long sequences that are much longer than conventional sequence transformation systems. As a result, the system can more efficiently perform sequence transformation tasks that require processing long input sequences, generating long output sequences, or both. For example, the system may outperform conventional systems on expressive summarization tasks that require generating long summaries of multiple documents. Such tasks and other long sequence transformation tasks may require processing and extracting information from input sequences that include 10,000 or more tokens to efficiently generate output sequences. However, because the system is entirely or primarily attention-based, the system is still as computationally efficient as the prior art, or in many cases more computationally efficient than the prior art.

[0010] Additionally, because the described system uses only a decoder neural network and does not require a separate encoder network, the number of parameters and therefore the memory consumed by storing and running inference using neural networks is greatly reduced relative to other systems that are able to perform well on sequence transformation tasks.

[0011] Furthermore, by utilizing local attention, memory compression attention, or both as described in this specification, the described system is able to efficiently perform sequence transformations on very long sequences without consuming excessive amounts of computational resources.

[0012] More generally, the described system also outperforms many existing systems due to the use of self-attention. Many existing approaches to sequence transformation using neural networks use recurrent neural networks in both the encoder and the decoder. While these kinds of networks can achieve good performance on sequence transformation tasks, their computation is inherently sequential, i.e., the recurrent neural network generates an output at the current time step that is conditioned on the hidden state of the recurrent neural network at the previous time step. This sequential nature prevents parallelization, leading to long training and inference times, and therefore workloads that use a lot of computational resources.

[0013] On the other hand, because the decoder of the described system is attention-based, the system can transform sequences faster, be trained faster, or both because the operation of the network can be more easily parallelized. That is, because the described neural network relies entirely on the attention mechanism to capture the global dependencies between inputs and outputs and does not employ any recurrent neural network layers, the problems of long training and interference times and high resource usage caused by the sequential nature of recurrent neural network layers are alleviated.

[0014] In addition, the described neural network can convert sequences more accurately than existing networks based on convolutional or recurrent layers, even with shorter training and inference times. In particular, in conventional models, the number of operations required to associate signals from two arbitrary input or output positions grows with the distance between the positions, for example, linearly or logarithmically, depending on the model architecture. This makes it more difficult to learn dependencies between distant positions during training. In the currently described neural network, the number of such operations is reduced to a constant number of operations due to the use of attention (and in particular self-attention) without relying on recurrence or convolution. Self-attention, sometimes also called internal attention, is an attention mechanism associated with different positions of a single sequence to facilitate the computation of a representation of the sequence. The use of an attention mechanism allows the neural network to effectively learn dependencies between positions at a distance during training, thereby improving the accuracy of the neural network on various conversion tasks, such as machine translation. The described neural network can also demonstrate performance that is superior to conventional sequence conversion neural networks without task-specific adjustments through the use of an attention mechanism.

[0015] The details of one or more embodiments of the subject matter described in this specification are set forth in the drawings and description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 An example neural network system is shown.

[0017] Figure 2A is a diagram showing the attention mechanism applied by the attention sub-layer in the sub-network of the decoder neural network.

[0018] Figure 2B is a diagram showing an attention mechanism that reduces the computational cost of processing long sequences.

[0019] Figure 3 is a flow chart of an example process for generating an output sequence from an input sequence.

[0020] Like reference numbers and numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0021] The present specification describes a system implemented as a computer program on one or more computers at one or more locations that generates a target sequence comprising a corresponding output at each of a plurality of positions in an output order from an input sequence comprising a corresponding input at each of a plurality of positions in an input order, i.e., converts an input sequence into a target sequence.

[0022] For example, the system can be a neural machine translation system. That is, if the input sequence is a sequence of words in the original language, such as a sentence or a phrase, the target sequence can be a translation of the input sequence into the target language, i.e., a sequence of words in the target language representing the sequence of words in the original language.

[0023] As another example, the system may be a speech recognition system. That is, if the input sequence is a sequence of audio data representing a spoken utterance, the target sequence may be a sequence of phonemes, characters or words representing the utterance, i.e., a transcription of the input sequence.

[0024] As another example, the system may be a natural language processing system. For example, if the input sequence is a sequence of words in the original language, such as a sentence or phrase, the target sequence may be a summary of the input sequence in the original language, i.e., a sequence having fewer words than the input sequence but retaining the basic meaning of the input sequence. As another example, if the input sequence is a sequence of words forming a question, the target sequence may be a sequence of words forming an answer to the question.

[0025] As another example, the system can be part of a computer-aided medical diagnosis system. For example, the input sequence can be a data sequence from an electronic medical record, and the target sequence can be a predicted treatment sequence.

[0026] As another example, the system may be part of an image processing system. For example, the input sequence may be an image, i.e., a sequence of color values ​​from an image, and the output may be a sequence of text describing the image. As another example, the input sequence may be a sequence of text or a different context, and the output sequence may be an image describing the context.

[0027] As another example, the system may be part of an extractive summarization system. In particular, the input sequence may be text from a plurality of input documents, and optionally, may be the topics of the documents, and the output sequence may be a textual summary of the input documents.

[0028] In particular, the neural network is a self-attention based decoder neural network. In some cases, the decoder does not include any convolutional layers or any recurrent layers.

[0029] Figure 1 An exemplary neural network system 100 is shown. Neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.

[0030] Neural network system 100 receives input sequence 102 and processes input sequence 102 to transform input sequence 102 into output sequence 152 .

[0031] The input sequence 102 has a corresponding input token at each of the plurality of input positions in the input order, and the output sequence 152 has a corresponding output token at each of the plurality of output positions in the output order. That is, the input sequence 102 has a plurality of inputs arranged according to the input order, and the output sequence 152 has a plurality of outputs arranged according to the output order.

[0032] As described above, neural network system 100 may perform any of a variety of tasks that require processing sequential inputs to generate sequential outputs. In a specific example where neural network system 100 performs expressive summarization, the input sequence may include text from multiple documents, and the output sequence may be text summarizing the multiple documents. Optionally, the input sequence may also include, for example, at the beginning of the input sequence, a desired topic for the summary text, i.e., text specifying the topic to which the multiple documents relate.

[0033] The neural network system 100 includes a self-attention decoder neural network 150. As will be described in more detail below, the self-attention decoder neural network 150 includes a plurality of neural network layers including a plurality of masked self-attention neural network layers.

[0034] The decoder neural network 150 is configured to generate output sequences in an autoregressive manner.

[0035] That is, the decoder neural network 150 generates an output sequence outputted by outputting by generating an output token at a corresponding output position at each of a plurality of generation time steps. That is, at each generation time step, the decoder neural network 150 generates a new output token at the next output position in the output order conditioned on the input sequence, and an output token at the output position before the next output position in the output order.

[0036] In particular, for a given output position, the decoder neural network 150 generates a time step output that defines a probability distribution over possible output tokens at the given output position.

[0037] The system 100 can then select the network output for that output position by sampling from the probability distribution or by selecting the output token with the highest probability.

[0038] More specifically, at each generation time step, the system 100 generates a combined sequence 108 for that generation time step.

[0039] The combined sequence 108 includes the input sequence, followed by the output token that has been generated up to the generation time step, that is, the output token at the previous position in the output order. In some embodiments, the output token that has been generated immediately follows the input sequence token in the combined sequence 108. In some other embodiments, the input sequence and the output token that has been generated up to the generation time step are separated by a predetermined specific separator token in the combined sequence 108.

[0040] In other words, the system 100 represents the input sequence and the already generated output together as a single combined sequence, eliminating the need to employ an encoder neural network during the transformation of the input sequence.

[0041] The decoder neural network 150 then processes the combined sequence 108 to generate an output that defines a probability distribution over possible output tokens at the output position.

[0042] Because the decoder neural network 150 is autoregressive, at each generation time step, the decoder 150 operates on output tokens that have been generated before the generation time step - i.e., outputs at output positions that precede the corresponding output position in the output order. In some embodiments, to ensure that this is the case both during inference and training, at each generation time step, the decoder neural network 150 shifts the already generated outputs to the right by one output order position (i.e., introduces a position shift to the already generated network output sequence) and (as will be described in more detail below) masks certain operations so that positions only pay attention to positions up to and including that position in the output sequence (and not to subsequent positions). Although the remainder of the description below describes that, when generating a given output at a given output position, the various components of the decoder 150 operate on data at output positions that precede the given output position (and not on data at any other output position), it will be appreciated that this type of conditioning can be efficiently implemented using the above-described shifting.

[0043] The decoder neural network 150 includes an embedding layer 120, a sequence of one or more decoder subnetworks 130, a linear layer 180, and a softmax layer 190. In particular, Figure 1 As shown in , the decoder neural network includes N decoder sub-networks 130.

[0044] The embedding layer 120 is configured to map the token to a digital representation of the token in the embedding space, for example, to a vector in the embedding space, for each token in the combined sequence. The embedding layer 120 then provides the digital representation of the token to the first subnetwork in the sequence of decoder subnetworks 130, i.e., to the first decoder subnetwork 130 in the N decoder subnetworks 130.

[0045] In particular, in some embodiments, the embedding layer 120 is configured to map each token to an embedding representation of the network input, and then combine, e.g., sum, average, or concatenate, the embedding representation of the token with the position embedding of the position of the token in the combined sequence to generate a combined embedding representation of the token. That is, each position in the combined sequence has a corresponding embedding, and for each token, the embedding layer 120 combines the embedding representation of the token with the embedding of the token position in the combined sequence.

[0046] In some cases, the position embedding is learned. As used in this specification, the term "learned" means that the operation or value has been adjusted during the training of the decoder neural network 150. Figure 3 Training the decoder neural network 150 is described.

[0047] In some other cases, the position embedding is fixed and different for each position. For example, the embedding can be composed of sine and cosine functions of different frequencies and can satisfy:

[0048]

[0049] where pos is the position, i is the dimension within the position embedding, and d model is the dimensionality of the position embedding (and other vectors processed by the neural network 150).

[0050] The combined embedding representation is then used as the numeric representation of the token.

[0051] Each decoder sub-network 130 is configured to receive a corresponding decoder sub-network input for each of a plurality of combined sequence positions and to generate a corresponding sub-network output for each of a plurality of combined sequence positions.

[0052] The decoder subnetwork output generated by the last decoder subnetwork in the sequence is then provided as input to the linear layer 180.

[0053] For the first decoder sub-network in the sequence, the decoder sub-network input is the digital representation generated by the embedding layer 120, and for each decoder sub-network except the first decoder sub-network in the sequence, the decoder sub-network input is the decoder sub-network output of the previous decoder sub-network in the sequence.

[0054] Each decoder sub-network 130 includes a decoder masked self-attention sub-layer 132. The decoder self-attention sub-layer 132 is configured to receive, at each generation time step, the input of each combined sequence position before the corresponding output position, i.e., before the output position for which the output token is currently being generated, and, for each particular combined sequence position, apply an attention mechanism to the input at the combined sequence position before the corresponding position using one or more queries derived from the input at the particular output position to generate an updated representation for the particular output position.

[0055] That is, the decoder applies a masked attention mechanism from the attention sub-layer 172 so that it does not pay attention to or otherwise process any data at positions in the combined sequence that do not precede the current output position.

[0056] The masked attention mechanism and how the decoder self-attention sublayer 132 applies the attention mechanism will be described in more detail below with reference to Figure 2.

[0057] In some examples, different decoder self-attention sublayers 132 in different decoder subnetworks 130 employ different attention mechanisms. For example, as will be described below with reference to Figure 2A and Figure 2B As described, some self-attention sub-layers may employ local attention while others employ memory-compressed attention. In particular, in some embodiments, the type of attention alternates between sub-networks, i.e., every other sub-network employs memory-compressed attention and the rest of the sub-networks use local attention.

[0058] In some embodiments, each decoder sub-network 130 further includes a residual connection layer that combines the output of the decoder self-attention sub-layer with the input to the decoder self-attention sub-layer to generate a decoder self-attention residual output and a layer normalization layer that applies layer normalization to the decoder self-attention residual output. Figure 1 They are collectively referred to as "Add&Norm" operations.

[0059] Some or all of the decoder subnetworks may also include a position-by-position feed-forward layer 134 that is configured to operate individually on each position in the combined sequence. In particular, for each combined sequence position, the feed-forward layer 134 is configured to receive an input at the combined sequence position and apply a sequence of transformations to the input at the combined sequence position to generate an output for the combined sequence position. For example, the sequence of transformations may include two or more learned linear transformations, each separated by an activation function, such as a nonlinear element-wise activation function, such as a ReLU activation function. The input received by the position-by-position feed-forward layer 134 may be the output of a layer normalization layer when a residual and layer normalization layer is included, or the output of a decoder self-attention sublayer 132 when a residual and layer normalization layer is not included. For each input position, the transformation applied by the layer 134 will typically be the same (but different feed-forward layers in different subnetworks will apply different transformations).

[0060] Where the decoder subnetwork 130 includes a position-by-position feedforward layer 134, the decoder subnetwork may also include: a residual connection layer that combines the output of the position-by-position feedforward layer with the input to the position-by-position feedforward layer to generate a decoder position-by-position residual output; and a layer normalization layer that applies layer normalization to the decoder position-by-position residual output. Figure 1 In the above example, these two layers are also collectively referred to as “Add&Norm” operations. The output of this normalization layer can then be used as the output of the decoder sub-network 130.

[0061] At each generation time step, the linear layer 180 applies the learned linear transformation to the output of the last decoder sub-network 130 so as to project the output of the last decoder sub-network 130 into an appropriate space for processing by the softmax layer 190. The softmax layer 190 then applies a softmax function on the output of the linear layer 180 to generate a probability distribution over possible network outputs at the generation time step.

[0062] If the selected output token at a given generation time step is a predetermined end-of-sequence token, the system 100 may determine that the output sequence 152 is complete and provide the output tokens that have been generated up to that generation time step as the final output sequence 152 of the input sequence 102 .

[0063] Therefore, from Figure 1As can be seen in FIG, rather than relying on an encoder neural network to first encode an input sequence and then generate an output sequence conditioned on the encoding of the input sequence, as occurs in some conventional systems, system 100 operates on the input sequence by combining sequences at each generation time step. This use of combined sequences allows system 100 to process long input sequences efficiently, i.e., because information from the entire input sequence is directly available at each generation time step, while still having fewer parameters than conventional systems, i.e., because system 100 does not have an encoder neural network.

[0064] Despite Figure 1 Not shown, but in some cases, to increase the computational power of the decoder neural network 150 without unduly increasing processing time or computational cost, the decoder neural network 150 may include one or more mixture-of-experts layers. At each time step, the mixture-of-experts layer selects a small subset of a large number of experts and combines the outputs from the small subset of experts to generate a layer output for the time step. The mixture of experts layer is described in more detail in Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017.

[0065] Figure 2A is a diagram 200 showing the attention mechanism applied by the attention sub-layer in the sub-network of the decoder neural network 150.

[0066] In general, an attention mechanism maps a query and a set of key-value pairs to an output, where the query, key, and value are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is calculated by a compatibility function between the query and the corresponding key.

[0067] More specifically, each attention sublayer applies a scaled dot product attention mechanism. In scaled dot product attention, for a given query, the attention sublayer computes the dot products of the query and all keys, divides each dot product by a scaling factor, e.g., by the square root of the dimension of the query and keys, and then applies a softmax function on the scaled dot products to obtain a weight for the value. The attention sublayer then computes a weighted sum of the values ​​based on these weights. Thus, for scaled dot product attention, the compatibility function is the dot product, and the output of the compatibility function is further scaled by the scaling factor.

[0068] In operation and as shown in the left-hand side of Figure 2, the attention sublayer simultaneously computes attention on a set of queries. In particular, the attention sublayer packs the queries into a matrix Q, the keys into a matrix K, and the values ​​into a matrix V. To pack a set of vectors into a matrix, the attention sublayer may generate a matrix containing the vectors as matrix rows.

[0069] The attention sublayer then performs a matrix multiplication between the matrix Q and the transpose of the matrix K to produce the matrix output by the compatibility function.

[0070] The attention sublayer then scales the compatibility function output matrix, i.e., by dividing each element of the matrix by the scaling factor.

[0071] The attention sublayer then applies softmax on the scaled output matrix to generate a weight matrix, and performs matrix multiplication between the weight matrix and the matrix V to generate an output matrix including the output of the attention mechanism for each value.

[0072] Because the decoder attention sublayer uses masks, the attention sublayer masks the scaled output matrix before applying the softmax. That is, the attention sublayer masks (sets to negative infinity) all values ​​in the scaled output matrix corresponding to positions after the current output position.

[0073] In some embodiments, in order to allow the attention sub-layer to jointly attend to information from different representation subspaces at different positions, the attention sub-layer adopts multi-head attention, as illustrated in the right-hand side of Figure 2.

[0074] In particular, to implement multi-head attention, the attention sublayer applies h different attention mechanisms in parallel. In other words, the attention sublayer includes h different attention layers, and each attention layer in the same attention sublayer receives the same original query Q, original key K, and original value V.

[0075] Each attention layer is configured to transform the original query, key, and value using a learned linear transformation, and then apply the attention mechanism to the transformed query, key, and value. Each attention layer will typically learn a different transformation than every other attention layer in the same attention sublayer.

[0076] In particular, each attention layer is configured to apply the learned query linear transformation to each original query to generate a layer-specific query for each original query, apply the learned key linear transformation to each original key to generate a layer-specific key for each original key, and apply the learned key linear transformation to each original value to generate a layer-specific value for each original value. The attention layer then applies the above-mentioned attention mechanism using these layer-specific queries, keys, and values ​​to generate the initial output of the attention layer.

[0077] The attention sub-layer then combines the initial outputs of the attention layer to generate the final output of the attention sub-layer. As shown in Figure 2, the attention sub-layer concatenates the outputs of the attention layer and applies the learned linear transformation to the concatenated outputs to generate the output of the attention sub-layer.

[0078] In some cases, the learned transformation applied by the attention sublayer reduces the dimensionality of the original keys and values, and optionally the query. For example, when the dimensionality of the original keys, values, and queries is d and there are h attention layers in the sublayer, the sublayer can reduce the dimensionality of the original keys, values, and queries to d / h. This keeps the computational cost of the multi-head attention mechanism similar to the cost of performing the attention mechanism once with the full dimensionality, while increasing the representational power of the attention sublayer.

[0079] For each decoder self-attention sublayer, each position in the decoder attends to all positions in the decoder that preceded that position. Hence, all keys, values, and queries come from the same position, and in this case, the output of the previous subnetwork in the decoder, or for the decoder self-attention sublayer in the first decoder subnetwork, the embeddings of the combined sequence tokens have already been generated. Hence, there is a corresponding key, value, and query for each position that precedes the current position in output order.

[0080] When the decoder self-attention sublayer implements multi-head attention, each attention layer in the decoder self-attention sublayer is configured to, at each generation time step, apply the learned query linear transformation to the input at each combined sequence position before the corresponding output position to generate a corresponding query for each combined sequence position, apply the learned key linear transformation to each input at each combined sequence position before the corresponding output position to generate a corresponding key for each output position, apply the learned value linear transformation to each input at each combined sequence position before the corresponding output position to generate a corresponding key for each output position, and then apply an attention mechanism (i.e., the scaled dot product attention mechanism described above) using the query, key, and value to determine an initial decoder self-attention output for each output position. The sublayer then combines the initial outputs of the attention layers as described above.

[0081] In some embodiments, to allow the decoder neural network to process long combinatorial sequences more efficiently, the attention mechanism is modified by limiting the dot product between Q and K in order to reduce the computational cost of the attention mechanism while maintaining high-quality output generation.

[0082] Figure 2B is an example diagram showing an attention mechanism that reduces the computational cost of processing long sequences.

[0083] In particular, Figure 2B Three different types of attention mechanisms are shown: the masked multi-head attention mechanism 250 described above, memory compression attention 270, and local attention 290.

[0084] As described above, for the masked multi-headed attention 250, the matrices V, K, and Q are determined, and then the masked multi-headed attention is applied.

[0085] On the other hand, in memory compression attention 270, after determining the matrices V, K, and Q as described above, the number of keys and values ​​is reduced by applying strided convolution ("Conv") to the V and K matrices. The number of queries in the Q matrix remains unchanged. This modification divides the number of activations by a compression factor based on the kernel size and the strided convolution stride factor. For example, the size of the convolution kernel can be 3 and the stride of the strided convolution factor can be 3. Masked multi-head attention is then applied to the reduced V and K matrices as described above.

[0086] Therefore, due to the application of strided convolutions, the memory compression attention mechanism 270 is able to globally exchange information across the entire sequence at a reduced computational cost relative to the standard multi-head attention mechanism 250.

[0087] Unlike memory compression attention 270 and multi-head attention 250, local attention 290 performs attention independently within each of the multiple blocks of the combined sequence. In particular, for local attention, the combined sequence tokens are divided into blocks (or "subsequences") of similar length, and attention is performed independently in each block. Because the storage cost of attention per block is constant in this scheme, this modification keeps the number of activations linear with respect to the sequence length, but does not allow each position to pay attention to the entire sequence. In more detail, for local attention, the input sequence to the attention mechanism is split into blocks ("split"), and multi-head attention as described above is applied independently within each block. The attention outputs of each block are then concatenated ("merged") to generate the output of the local attention mechanism.

[0088] As described above, in some embodiments, different decoder self-attention sublayers 132 in different decoder subnetworks 130 employ different attention mechanisms. For example, some self-attention sublayers may employ local attention while others employ memory compression attention. In particular, in some embodiments, the type of attention alternates between subnetworks, i.e., every other, i.e., every other, subnetwork employs memory compression attention while the remaining subnetworks employ local attention. By having some layers apply memory compression attention while other layers employ local attention, the decoder neural network can attend to the entire combined sequence without consuming an excessive amount of computational resources, even when the combined sequence is long.

[0089] Figure 3 is a flow chart of an example process for generating an output sequence from an input sequence. For convenience, process 300 is described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system, such as one appropriately programmed according to this specification Figure 1 The neural network system 100 can perform process 300.

[0090] The system can perform process 300 at each of a plurality of generation time steps to generate an output sequence using a self-attention decoder neural network. The decoder neural network is configured to generate an output sequence from an input sequence in an autoregressive manner. That is, the decoder neural network generates one output from the output sequence at each generation time step. Thus, by performing process 300 at each of a plurality of generation time steps, the system generates all output tokens in the output sequence.

[0091] The system generates a combined sequence for a generation time step that includes the input sequence followed by the output tokens that have been generated up to that generation time step (step 310).

[0092] The system processes the combined sequence using a self-attention decoder neural network to generate a time-step output that defines a distribution of scores over a set of possible output tokens (step 320).

[0093] The system uses the time step output to select an output token from the set of possible output tokens as the next output token in the output sequence (step 330).

[0094] The system may perform process 300 for an input sequence for which the desired output, ie, the output sequence that the system should generate for the input sequence, is unknown. In this case, the system sequentially performs process 300 for all generation time steps.

[0095] The system can also perform process 300 on input sequences in a training data set, i.e., the input set for which the system should generate output sequences is known, so as to facilitate training the decoder to determine training values ​​for the parameters of the decoder. Process 300 can be repeatedly performed on inputs selected from the training data set as part of conventional machine learning training techniques to train initial neural network layers, such as gradient descent using a back-propagation training technique using a conventional optimizer, such as an Adam optimizer. During the training process, the system can include a variety of techniques to improve the speed, effectiveness, or both of the training process. For example, the system can use dropout, label smoothing, or both to reduce overfitting. As another example, the system can perform training using a distributed architecture that trains multiple instances of a sequence transformation neural network in parallel.

[0096] However, during training, since the entire ground truth output sequence is available at the beginning of the training process, the system can generate time-step outputs in parallel for each generation time step, i.e., the system does not need to perform process 300 sequentially for each generation time step. In other words, the system can provide as input to the decoder neural network a combined sequence including an input sequence followed by an entire known output sequence, and can generate time-step outputs in parallel for all generation time steps. The system can then update the decoder parameters using machine learning training techniques to optimize a maximum likelihood objective that depends on the score assigned to the correct output token at each time step.

[0097] Additionally, in some embodiments, during training, the system also includes the prediction of the next token in the input sequence in the maximum likelihood objective. In other words, during training, the decoder also generates, for each input position, a distribution of scores for possible input tokens conditioned on the input token at the previous position in the input sequence. The system then includes the score assigned to the correct input token at each input position in the maximum likelihood objective. In doing so, the model is forced to accurately predict the next token in the input as well as the next token in the output, and error signals are propagated from the input and output positions during training, thereby improving the performance of the model after it is trained.

[0098] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination of software, firmware, hardware that, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0099] Embodiments of the subject matter and functional operations described in this specification may be implemented with digital electronic circuits, with tangibly implemented computer software or firmware, with computer hardware including the structures disclosed in this specification and their structural equivalents, or with a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0100] The term "data processing apparatus" refers to data processing hardware and includes all kinds of apparatus, devices and machines for processing data, including by way of example a programmable processor, a computer or a plurality of processors or computers. An apparatus may also be or further include special-purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). An apparatus may optionally include, in addition to hardware, code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0101] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language including compiled or interpreted languages ​​or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document; in a single file dedicated to the program or in multiple coordinated files, such as files storing one or more modules, subroutines or portions of code. A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0102] In this specification, the term "database" is used broadly to refer to any data set: the data need not be structured in any particular way, or at all, and it may be stored on storage devices in one or more locations. Thus, for example, an index database may include multiple data sets, each of which may be organized and accessed differently.

[0103] Similarly, the term "engine" is used broadly in this specification to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed and run on the same computer or multiple computers.

[0104] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit such as an FPGA or ASIC, or by a combination of a special purpose logic circuit and one or more programmed computers.

[0105] A computer suitable for executing a computer program may be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or any other kind of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing or implementing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory may be supplemented by a dedicated logic circuit or incorporated in the dedicated logic circuit. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or may be operationally coupled to receive data from the one or more large-capacity storage devices or to transfer data to the one or more large-capacity storage devices, or both for storing data. However, a computer need not have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game controller, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0106] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including by way of example semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0107] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to receiving a request from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device, such as a smart phone running a messaging application, and then receiving a response message from the user.

[0108] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production, i.e., inference, workloads.

[0109] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0110] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server; or includes a middleware component, such as an application server; or includes a front-end component, such as a client computer with a graphical user interface, a web browser, or an app that a user can use to interact with an implementation of the subject matter described in this specification; or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0111] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data such as HTML pages to the user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from the user. Data generated at the user device, for example, the result of the user interaction, may be received from the device at the server.

[0112] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. In addition, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may be removed from the combination in some cases, and a claimed combination may be directed to a subcombination or variations of a subcombination.

[0113] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0114] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims may be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method of generating an output sequence comprising a plurality of output tokens from an input sequence comprising a plurality of input tokens, the method comprising, at each generation time step of a plurality of generation time steps: generating a combined sequence for the generation time step, the combined sequence comprising the input sequence followed by output tokens generated up to the generation time step, wherein the combined sequence is a single combined sequence for collectively processing the input sequence and the output tokens generated up to the generation time step; The combined sequence is processed using a self-attention decoder neural network, where the self-attention decoder neural network comprising a plurality of neural network layers, the plurality of neural network layers comprising a plurality of masked self-attention neural network layers, wherein the self-attention decoder neural network is configured to process the combined sequence through the plurality of neural network layers to generate a time step output, the time step output defining a distribution of scores over a set of possible output tokens, and wherein the plurality of masked self-attention neural network layers are masked such that the time step output depends only on the input sequence and output tokens that have been generated up to that generation time step, and does not depend on any output token after a last token that has been generated in the output sequence; as well as An output token is selected from the set of possible output tokens using the time step output as the next output token in the output sequence.

2. The method according to claim 1, wherein: The input sequence and the output tokens that have been generated up to the generation time step are separated by a predetermined special separator token in the combined sequence.

3. The method according to claim 1, wherein: The multiple masked self-attention neural network layers are masked multi-head attention layers.

4. The method according to claim 1, wherein: The plurality of masked self-attention neural network layers comprises at least one local attention layer, and wherein each local attention layer comprises a local attention sub-layer, the local attention sub-layer being configured to: receiving a layer input sequence comprising a plurality of layer inputs; Dividing the layer input sequence into a plurality of subsequences having similar lengths; For each subsequence in the plurality of subsequences, generating a subsequence output by performing self-attention on the layer inputs in the subsequence; and The subsequence outputs are merged to generate a layer output sequence.

5. The method according to claim 1, wherein: The plurality of masked self-attention neural network layers comprises at least one memory compression attention layer, and wherein each memory compression attention layer comprises a memory compression sublayer, the memory compression sublayer being configured to: Get an attention input including multiple keys, values, and queries; applying a strided convolution to the keys to generate a reduced set of keys; applying a strided convolution to the values ​​to generate a reduced set of values; A layer output sequence is generated by performing self-attention using the reduced set of keys, the reduced set of values, and the query.

6. The method according to claim 5, wherein: Obtaining the attention input includes: receiving a layer input sequence comprising a plurality of layer inputs; and The layer input sequence is projected into the key, the value, and the query using a corresponding projection matrix.

7. The method according to claim 1, wherein: The input sequence includes text from a plurality of documents, and wherein the output sequence is text summarizing the plurality of documents.

8. The method according to claim 7, wherein: The input sequence also includes text specifying topics to which the plurality of documents relate.

9. The method according to claim 1, further comprising: determining that the selected output for the time step is a predetermined end-of-sequence token; and In response, the output tokens that have been generated up to the generation time step are provided as a final output sequence of the input sequence.

10. The method according to any one of claims 1 to 9, wherein: The plurality of neural network layers include one or more mixture-of-experts layers.

11. A system comprising one or more computers and one or more storage devices storing instructions, which when executed by the one or more computers cause the one or more computers to perform the operations of the method according to any one of claims 1-10.

12. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1-10.

13. A computer-implemented method for generating an output sequence comprising a plurality of output tokens from an input sequence comprising a plurality of input tokens, wherein the input sequence comprises an image and the output sequence comprises a text sequence describing the image, the method comprising, at each of a plurality of generation time steps: generating a combined sequence for the generation time step, the combined sequence comprising the input sequence followed by output tokens generated up to the generation time step, wherein the combined sequence is a single combined sequence for collectively processing the input sequence and the output tokens generated up to the generation time step; The combined sequence is processed using a self-attention decoder neural network, where the self-attention decoder neural network comprising a plurality of neural network layers, the plurality of neural network layers comprising a plurality of masked self-attention neural network layers, wherein the self-attention decoder neural network is configured to process the combined sequence through the plurality of neural network layers to generate a time step output, the time step output defining a distribution of scores over a set of possible output tokens, and wherein the plurality of masked self-attention neural network layers are masked such that the time step output depends only on the input sequence and output tokens that have been generated up to that generation time step, and does not depend on any output token after a last token that has been generated in the output sequence; as well as An output token is selected from the set of possible output tokens using the time step output as the next output token in the output sequence.

14. The method according to claim 13, wherein: The input sequence and the output tokens that have been generated up to the generation time step are separated by a predetermined special separator token in the combined sequence.

15. The method according to claim 13, wherein: The multiple masked self-attention neural network layers are masked multi-head attention layers.

16. The method according to claim 13, wherein: The plurality of masked self-attention neural network layers comprises at least one local attention layer, and wherein each local attention layer comprises a local attention sub-layer, the local attention sub-layer being configured to: receiving a layer input sequence comprising a plurality of layer inputs; Dividing the layer input sequence into a plurality of subsequences having similar lengths; For each subsequence in the plurality of subsequences, generating a subsequence output by performing self-attention on the layer inputs in the subsequence; and The subsequence outputs are merged to generate a layer output sequence.

17. The method according to claim 13, wherein: The plurality of masked self-attention neural network layers comprises at least one memory compression attention layer, and wherein each memory compression attention layer comprises a memory compression sublayer, the memory compression sublayer being configured to: Get an attention input including multiple keys, values, and queries; applying a strided convolution to the keys to generate a reduced set of keys; applying a strided convolution to the values ​​to generate a reduced set of values; A layer output sequence is generated by performing self-attention using the reduced set of keys, the reduced set of values, and the query.

18. The method according to claim 17, wherein: Obtaining the attention input includes: receiving a layer input sequence comprising a plurality of layer inputs; and The layer input sequence is projected into the key, the value, and the query using a corresponding projection matrix.

19. The method according to claim 13, further comprising: determining that the selected output for the time step is a predetermined end-of-sequence token; and In response, the output tokens that have been generated up to the generation time step are provided as a final output sequence of the input sequence.

20. The method according to any one of claims 13 to 19, wherein: The plurality of neural network layers include one or more mixture-of-experts layers.

21. A computer-implemented method of generating an output sequence comprising a plurality of output tokens from an input sequence comprising a plurality of input tokens, wherein the input sequence comprises a text sequence and the output sequence comprises an image describing a context associated with the text sequence, the method comprising, at each of a plurality of generation time steps: generating a combined sequence for the generation time step, the combined sequence comprising the input sequence followed by output tokens generated up to the generation time step, wherein the combined sequence is a single combined sequence for collectively processing the input sequence and the output tokens generated up to the generation time step; The combined sequence is processed using a self-attention decoder neural network, where the self-attention decoder neural network comprising a plurality of neural network layers, the plurality of neural network layers comprising a plurality of masked self-attention neural network layers, wherein the self-attention decoder neural network is configured to process the combined sequence through the plurality of neural network layers to generate a time step output, the time step output defining a distribution of scores over a set of possible output tokens, and wherein the plurality of masked self-attention neural network layers are masked such that the time step output depends only on the input sequence and output tokens that have been generated up to that generation time step, and does not depend on any output token after a last token that has been generated in the output sequence; as well as An output token is selected from the set of possible output tokens using the time step output as the next output token in the output sequence.

22. The method according to claim 21, wherein: The input sequence and the output tokens that have been generated up to the generation time step are separated by a predetermined special separator token in the combined sequence.

23. The method according to claim 21, wherein: The multiple masked self-attention neural network layers are masked multi-head attention layers.

24. The method according to claim 21, wherein: The plurality of masked self-attention neural network layers comprises at least one local attention layer, and wherein each local attention layer comprises a local attention sub-layer, the local attention sub-layer being configured to: receiving a layer input sequence comprising a plurality of layer inputs; Dividing the layer input sequence into a plurality of subsequences having similar lengths; For each subsequence in the plurality of subsequences, generating a subsequence output by performing self-attention on the layer inputs in the subsequence; and The subsequence outputs are merged to generate a layer output sequence.

25. The method according to claim 21, wherein: The plurality of masked self-attention neural network layers comprises at least one memory compression attention layer, and wherein each memory compression attention layer comprises a memory compression sublayer, the memory compression sublayer being configured to: Get an attention input including multiple keys, values, and queries; applying a strided convolution to the keys to generate a reduced set of keys; applying a strided convolution to the values ​​to generate a reduced set of values; A layer output sequence is generated by performing self-attention using the reduced set of keys, the reduced set of values, and the query.

26. The method according to claim 25, wherein: Obtaining the attention input includes: receiving a layer input sequence comprising a plurality of layer inputs; and The layer input sequence is projected into the key, the value, and the query using a corresponding projection matrix.

27. The method of claim 21, further comprising: determining that the selected output for the time step is a predetermined end-of-sequence token; and In response, the output tokens that have been generated up to the generation time step are provided as a final output sequence of the input sequence.

28. The method according to any one of claims 21 to 27, wherein: The plurality of neural network layers include one or more mixture-of-experts layers.

29. A computer-implemented method of generating an output sequence comprising a plurality of output tokens from an input sequence comprising a plurality of input tokens, wherein the input sequence comprises a sequence of audio data representing a spoken utterance and the output sequence comprises a sequence of phonemes, characters or words representing the spoken utterance, the method comprising, at each of a plurality of generation time steps: generating a combined sequence for the generation time step, the combined sequence comprising the input sequence followed by output tokens generated up to the generation time step, wherein the combined sequence is a single combined sequence for collectively processing the input sequence and the output tokens generated up to the generation time step; The combined sequence is processed using a self-attention decoder neural network, where the self-attention decoder neural network comprising a plurality of neural network layers, the plurality of neural network layers comprising a plurality of masked self-attention neural network layers, wherein the self-attention decoder neural network is configured to process the combined sequence through the plurality of neural network layers to generate a time step output, the time step output defining a distribution of scores over a set of possible output tokens, and wherein the plurality of masked self-attention neural network layers are masked such that the time step output depends only on the input sequence and output tokens that have been generated up to that generation time step, and does not depend on any output token after a last token that has been generated in the output sequence; as well as An output token is selected from the set of possible output tokens using the time step output as the next output token in the output sequence.

30. The method of claim 29, wherein: The input sequence and the output tokens that have been generated up to the generation time step are separated by a predetermined special separator token in the combined sequence.

31. The method of claim 29, wherein: The multiple masked self-attention neural network layers are masked multi-head attention layers.

32. The method of claim 29, wherein: The plurality of masked self-attention neural network layers comprises at least one local attention layer, and wherein each local attention layer comprises a local attention sub-layer, the local attention sub-layer being configured to: receiving a layer input sequence comprising a plurality of layer inputs; Dividing the layer input sequence into a plurality of subsequences having similar lengths; For each subsequence in the plurality of subsequences, generating a subsequence output by performing self-attention on the layer inputs in the subsequence; and The subsequence outputs are merged to generate a layer output sequence.

33. The method of claim 29, wherein: The plurality of masked self-attention neural network layers comprises at least one memory compression attention layer, and wherein each memory compression attention layer comprises a memory compression sublayer, the memory compression sublayer being configured to: Get an attention input including multiple keys, values, and queries; applying a strided convolution to the keys to generate a reduced set of keys; applying a strided convolution to the values ​​to generate a reduced set of values; A layer output sequence is generated by performing self-attention using the reduced set of keys, the reduced set of values, and the query.

34. The method of claim 33, wherein: Obtaining the attention input includes: receiving a layer input sequence comprising a plurality of layer inputs; and The layer input sequence is projected into the key, the value, and the query using a corresponding projection matrix.

35. The method of claim 29, further comprising: determining that the selected output for the time step is a predetermined end-of-sequence token; and In response, the output tokens that have been generated up to the generation time step are provided as a final output sequence of the input sequence.

36. A method according to any one of claims 29 to 35, wherein: The plurality of neural network layers include one or more mixture-of-experts layers.

37. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 13-36.

38. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 13-36.

39. A method for processing an input sequence to a decoder, wherein the input sequence is from a set of training data, wherein a ground truth output sequence for the input sequence is available, the method comprising: At each of multiple generation time steps: generating a combined sequence for the generation time step, the combined sequence comprising the input sequence followed by output tokens that have been generated up to the generation time step; processing the combined sequence using a self-attention decoder neural network to generate time-step outputs, the time-step outputs defining a distribution of scores over a set of possible output tokens; as well as selecting an output token from the set of possible output tokens as the next output token in the output sequence using the time step output, wherein time step outputs are generated in parallel for all of the plurality of generated time steps; Decoder parameters of the decoder are updated using machine learning training techniques to optimize a maximum likelihood objective that depends on the score assigned to the correct output token at each time step.

Citation Information

Patent Citations

  • Recurrent neural networks for data item generation

    CN105868829A

  • Augmenting neural networks with hierarchical external memory

    WO2017136081A1

Cited By

  • System and methods for mitigating attention noise in transformer models

    WO2026218295A1