Method for training transformer to improve training efficiency, and apparatus therefor

Abstract

The objective of the present invention is to provide a masking technique for enabling a transformer to simultaneously learn attention between input tokens combined in various lengths even with a small amount of training data so that all possible and meaningful relationships or compatibility can be considered in learning, and for enabling a trained transformer to have better performance. The present invention relates to a method performed in a computing device for training a neural network including n stacked encoder or decoder blocks (n is an integer of 1 or more), the method comprising the steps of: receiving a first input sequence including L tokens; generating a plurality of sub-sequences on the basis of L vectors respectively corresponding to the tokens of the first input sequence and generating a second input sequence by concatenating the plurality of sub-sequences; and performing a self-attention operation on the basis of the second input sequence.

Description

Transformer learning method and device for improving learning efficiency

[0001] The present invention relates to a neural network architecture, and more specifically, to a transformer, which is an attention-based sequence transforming neural network model.

[0002] Transformer, an attention-based sequence transforming neural network model proposed by Google in a 2017 paper titled "Attention Is All You Need," dramatically improved the problems of the previous RNN (Recurrent Neural Network) model, and since then, neural network technology using the attention mechanism has been widely used in various applications such as translation, text generation, chatbots, image generation, and music composition.

[0003] Recently, various language models have been announced, such as OpenAI's ChatGPT, Google's Bard, Meta's LLaMA, Stanford University's Alpaca, and LMSYS.org's Vicuna, and they are showing potential as artificial general intelligence that can be utilized in various fields as well as text generation.

[0004] Figure 1 illustrates an example of a prior art transformer neural network model. Transformer models come in various types, including those with both an encoder and decoder, as presented in a 2017 Google paper; those with only an encoder, like BERT; and those with only a decoder, like GPT and LLaMA. Figure 1 illustrates a structure with only an encoder or decoder.

[0005] The transformer model (100) illustrated in FIG. 1 may include various layers such as an input transducer (105), n stacked decoder (or encoder) blocks (110, 155, 160), a transposed embedding (165), and a softmax (170).

[0006] Here, each decoder / encoder block (110, 155, 160) may include layers such as a first Layer Norm (115), a Masked Multi-Headed Self-Attention (130), a first Add (135), a second Layer Norm (140), a Pointwise Feed Forward Network (145), and a second Add (150). Depending on the implementation, the functions of the Add and Norm layers may be combined to represent an Add & Norm layer, and the detailed configuration may be changed, such as omitting or adding some layers or changing their positions, as needed. For example, depending on the implementation, dropout layers (not shown) may be added to prevent over-fitting at necessary locations.

[0007] In Fig. 1, the input sequence Input X initmay be a sequence of input tokens, and the input transducer (105) generates an input token ID sequence by representing each token constituting the input sequence as a token ID, performs token embedding to convert it into an embedded vector corresponding to each token ID, and performs positional embedding to indicate the position of each token within the input sequence. Therefore, the input sequence Input X init Through this process, it is converted into an embedded vector sequence, and the embedded vector sequence is input to the first encoder / decoder block (110) as Input X in the form of a matrix.

[0008] Within each encoder / decoder block (110, 155, 160), self-attention operations are performed through a Masked Multi-Headed Self-Attention (130) layer, and a transformation process is performed through a Pointwise Feed Forward Network. In this process, the first Layer Norm (115) and the second Layer Norm (140) layers can perform normalization on the output of the previous layer at appropriate locations, and the first Add (135) and the second Add (150) layers can be employed at appropriate locations for residual connection.

[0009] The output generated by the final encoder / decoder block (160) through the stacked (the number of stacks n is appropriately determined according to the model) encoder / decoder blocks (110, 155, 160) is converted into a logit through an operation with the transposed matrix of the matrix used in the embedding performed by the input transducer (105) in the transposed embedding (165) layer, and based on this, probability is calculated in the softmax (170) layer to obtain a probability vector (175) for a token, and an output token (during inference) or loss (during training) is obtained.

[0010] Typically, natural language processing technologies use large-scale corpora, typically consisting of tens of gigabytes and hundreds of millions of tokens, for learning. A vocabulary is a set of tokens obtained by tokenizing all sentences and documents in the corpus and removing duplicates. In large language models such as transformer models, a dictionary for the encoder / decoder is constructed and referenced in the token embedding process performed in the input transducer (105) and the logit generation process performed in the transposed embedding (165) layer.

[0011] Transformer neural networks based on attention algorithms typically convert each token into a d-dimensional vector based on training data (corpus) consisting of thousands to trillions of sequential tokens, and then repeatedly extract several vectors from random locations and use them as input for learning hundreds to trillions of times.

[0012] However, as a practical issue, in many real-world applications, language models often fail to achieve the desired quality and performance (e.g., overfitting) due to insufficient training data. Therefore, a method to improve learning efficiency even with limited data is needed.

[0013] The present invention aims to propose a method for further improving the learning efficiency of a transformer even with a small amount of learning data.

[0014] More specifically, the present invention aims to achieve excellent learning performance with a smaller amount of data compared to the amount of corpus required for learning a conventional transformer.

[0015] Furthermore, the present invention enables a transformer to simultaneously learn attention between input tokens combined in various lengths even with a small amount of learning data, thereby allowing all possible and meaningful relevance or compatibility to be considered in learning.

[0016] Furthermore, the present invention proposes a masking technique that allows important relationships or correspondences between non-adjacent tokens (e.g., "if ~", "never ~", "not only ~ but also ~", etc.) that were not considered in the existing attention technique of the transformer to be better reflected in learning, while reducing the contribution to learning of unimportant relationships or correspondences, thereby enabling the learned transformer to have better performance.

[0017] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention pertains from the contents described in this specification.

[0018] In a first aspect of the present invention, a method for training a neural network including n stacked encoder or decoder blocks (wherein n is an integer greater than or equal to 1) is provided, the method being performed on a computing device, the method comprising: receiving a first input sequence including L tokens; generating a plurality of sub-sequences based on L vectors each corresponding to the tokens of the first input sequence, and concatenating the plurality of sub-sequences to generate a second input sequence; and performing a self-attention operation based on the second input sequence.

[0019] Here, the plurality of sub-sequences may each include at least the last vector among the L vectors.

[0020] Additionally, the plurality of sub-sequences may each have different lengths.

[0021] Additionally, at least some of the plurality of sub-sequences may include vectors each corresponding to a plurality of consecutive tokens on the first input sequence.

[0022] Additionally, the plurality of sub-sequences may be generated such that their lengths each increase by k (k is an integer greater than or equal to 1 and less than L).

[0023] Additionally, the order of the vectors included in each of the plurality of sub-sequences may be determined based on the order of each token in the first input sequence corresponding to each vector.

[0024] In addition, the concatenation of the plurality of sub-sequences may be performed by arranging the plurality of sub-sequences in increasing order of length from the shortest to the longest, according to the length of each sub-sequence.

[0025] In addition, at least one of the encoder or decoder blocks generates a query-key matrix to perform the self-attention operation, and performs a subsequent token masking process on the query-key matrix, wherein the subsequent token masking process comprises: at least one of the encoder or decoder blocks generates a query-key matrix to perform the self-attention operation, and performs a subsequent token masking process on the query-key matrix, wherein the subsequent token masking process comprises: for a matrix element of an i row / column and a j column / row of the query-key matrix, if a token of the first input sequence corresponding to a j-th vector of the second input sequence follows a token of the first input sequence corresponding to an i-th vector of the second input sequence on the first input sequence, replacing the matrix element with a predetermined value.

[0026] In addition, at least one of the encoder or decoder blocks may generate a query-key matrix to perform the self-attention operation, and perform a subsequent token masking process on the query-key matrix, wherein the subsequent token masking process may include replacing the matrix element of the i row / column and the j column / row of the query-key matrix with a predetermined value, if a set of vectors corresponding to tokens preceding a token corresponding to the j-th vector on the first input sequence, among vectors included in a sub-sequence to which the j-th vector of the second input sequence belongs, is not included in a set of vectors corresponding to tokens preceding a token corresponding to the i-th vector on the first input sequence, among vectors included in a sub-sequence to which the i-th vector of the second input sequence belongs.

[0027] In addition, at least one of the encoder or decoder blocks may generate a query-key matrix to perform the self-attention operation, and perform a subsequent token masking process on the query-key matrix, wherein the subsequent token masking process may include, when the matrix elements of a plurality of columns / rows among the matrix elements of i row / column of the query-key matrix have the same context, leaving only a predetermined number of the matrix elements of the plurality of columns / rows and replacing the rest with a predetermined value to avoid duplicate attention.

[0028] Additionally, the neural network may obtain a loss by using only an output corresponding to the last sub-sequence among the plurality of sub-sequences of the second input sequence among the output sequences processed through the stacked encoder or decoder blocks of the second input sequence.

[0029] In a third aspect of the present invention, a device is provided, comprising: a processor; and a memory, wherein the memory includes instructions configured to cause the device to implement a specific operation for training a neural network including n (wherein n is an integer greater than or equal to 1) stacked encoder or decoder blocks, wherein the specific operation may include: receiving a first input sequence including L tokens; generating a plurality of sub-sequences based on L vectors each corresponding to the tokens of the first input sequence, and concatenating the plurality of sub-sequences to generate a second input sequence; and performing a self-attention operation based on the second input sequence.

[0030] In a fourth aspect of the present invention, a computer-readable non-transitory storage medium is provided storing instructions configured to cause a device including the processor to implement a specific operation for training a neural network including n (wherein n is an integer greater than or equal to 1) stacked encoder or decoder blocks when executed by a processor, the specific operation including: receiving a first input sequence including L tokens; generating a plurality of sub-sequences based on L vectors each corresponding to the tokens of the first input sequence, and concatenating the plurality of sub-sequences to generate a second input sequence; and performing a self-attention operation based on the second input sequence.

[0031] According to the present invention, the learning efficiency of a transformer can be further improved even with a small amount of learning data.

[0032] More specifically, according to the present invention, it is possible to achieve excellent learning performance with a smaller amount of data compared to the amount of corpus required for learning a conventional transformer.

[0033] Furthermore, according to the present invention, the transformer can simultaneously learn attention between input tokens combined in various lengths even with a small amount of learning data, thereby allowing all possible and meaningful relevance or compatibility to be considered in learning.

[0034] Furthermore, according to the present invention, it is possible to achieve a masking technique that allows important relationships or correspondences between non-adjacent tokens (e.g., “if ~”, “never ~”, “not only ~ but also ~”) that were not considered in the existing attention technique of the transformer to be better reflected in learning, while reducing the contribution to learning of unimportant relationships or correspondences, thereby allowing the learned transformer to have better performance.

[0035] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the contents described in this specification.

[0036] The accompanying drawings are incorporated into and constitute a part of the detailed description to aid in the understanding of the present invention and together with the detailed description, illustrate embodiments and technical features of the present invention.

[0037] Figure 1 is a schematic diagram for exemplarily explaining the configuration and operation of a transformer neural network model of the prior art.

[0038] FIG. 2 is a conceptual diagram illustrating the configuration of an input transducer according to one embodiment of the present invention.

[0039] FIG. 3 is a conceptual diagram illustrating an incremental tails-concatenation process according to one embodiment of the present invention.

[0040] FIG. 4 is a flowchart illustrating an incremental tails-concatenation process according to one embodiment of the present invention.

[0041] FIG. 5 is a conceptual diagram illustrating a process of an improved masked multi-headed self-attention layer according to one embodiment of the present invention.

[0042] FIG. 6 is a diagram illustrating an improved Masking for Followings process according to one embodiment of the present invention.

[0043] FIG. 7 is a diagram illustrating an improved Masking for Followings process according to another embodiment of the present invention.

[0044] FIG. 8 is a diagram illustrating an improved Masking for Followings process according to another embodiment of the present invention.

[0045] Figure 9 illustrates an apparatus capable of performing the proposed method of the present invention.

[0046]

[0047] The present invention can be modified in various ways and has various embodiments. Hereinafter, specific embodiments will be described in detail based on the attached drawings.

[0048] The following examples are provided to provide a comprehensive understanding of the methods, devices, systems, and / or storage media described herein. However, these are merely examples and the scope of the present invention is not limited thereto.

[0049] In describing embodiments of the present invention, if a detailed description of a known technology related to the present invention is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of their functions in the present invention, and this may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terms used in the detailed description are only for the purpose of describing embodiments of the present invention and should never be construed as limiting. Unless clearly used otherwise, the singular form includes the plural form. In this description, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts or combinations thereof other than those described.

[0050] Additionally, although terms such as first, second, etc. may be used to describe various components, the components are not limited by the terms, and the terms are used only for the purpose of distinguishing one component from another.

[0051] Subsequence generation and incremental tail-concatenating

[0052] In an embodiment of the present invention, in order to train a neural network such as FIG. 1 including n (where n is an integer greater than or equal to 1) stacked encoder or decoder blocks, a first input sequence including L tokens is input, a plurality of sub-sequences are generated based on L vectors each corresponding to the tokens of the first input sequence, the plurality of sub-sequences are concatenated to generate a second input sequence, and a self-attention operation is performed based on the generated second input sequence.

[0053] FIG. 2 is a conceptual diagram illustrating the configuration of an input transducer (200) according to one embodiment of the present invention. In addition, FIG. 3 is a conceptual diagram illustrating an incremental tails-concatenating (220) process according to one embodiment of the present invention. Furthermore, FIG. 4 is a flowchart illustrating an incremental tails-concatenating (220) process according to one embodiment of the present invention.

[0054] As a more specific embodiment, in order to train a neural network including n (where n is an integer greater than or equal to 1) stacked encoder or decoder blocks, as illustrated in FIG. 4, a first input sequence including L tokens is input (S10), and based on L vectors each corresponding to the tokens of the first input sequence, a plurality of sub-sequences each including at least the last vector among the L vectors and each having a different length are generated (S20), and a second input sequence can be generated by concatenating the plurality of sub-sequences (S30). The inventors of the present invention refer to the plurality of sub-sequences each including at least the last vector among the above-described L vectors and each having a different length as tail sequences. According to an embodiment, at least some of the plurality of sub-sequences may include vectors each corresponding to a plurality of consecutive tokens on the first input sequence.

[0055] In Fig. 2, the input sequence Input X init may be a sequence of input tokens containing L tokens (corresponding to the first input sequence described above), and the improved input transducer (200) illustrated is an input sequence Input X init Each token that constitutes the input sequence is represented by a token ID, and the embedding (220) module performs token embedding, which converts each token ID into its corresponding embedded vector, and positional embedding, which indicates the position of each token in the input sequence, as in the prior art. Input sequence Input X init Through this process, it is converted into an embedded vector sequence, and an embedded vector sequence X(embedded) in the form of a matrix is ​​obtained.

[0056] As shown in Fig. 3, the embedded vector sequence X(embedded)(400) in the form of a matrix is ​​composed of L embedded vectors (X1, …, X L ) are included in each row, and each vector is an input sequence Input X init corresponds to each token in the image. In the example shown in Fig. 3, the case where the embedding dimension (or size) d is 2 is exemplified. In some embodiments, L embedded vectors (X1, …, X L ) can also be configured to include each of the columns.

[0057] According to an embodiment of the present invention, the improved input transducer (200) of FIG. 2 may further include an incremental tails-concatenating (220) module. Here, the incremental tails-concatenating (220) module is configured to concatenate L embedded vectors (X1, …, X) of an embedded vector sequence X (embedded) (400). L ) can generate a plurality of sub-sequences (420, 430, 440, 450, 460) and concatenate them to generate a vector sequence X (tails-concatenated) (410) (corresponding to the second input sequence described above). Each of the plurality of sub-sequences (420, 430, 440, 450, 460) can be generated as one of L embedded vectors (X1, …, X) in the embedded vector sequence X (embedded) (400). L ) the last vector X L, and may be generated to have different lengths. More specifically, each of the plurality of sub-sequences (420, 430, 440, 450, 460) may be generated to increase in length by k (k is an integer greater than or equal to 1 and less than L). In addition, the order of the vectors included in each of the sub-sequences (420, 430, 440, 450, 460) may be determined based on the order in the embedded vector sequence X (embedded) (400). In addition, at least some of the plurality of sub-sequences may include vectors respectively corresponding to a plurality of consecutive tokens on the first input sequence. In addition, the vector sequence X (tails-concatenated) (410) obtained by concatenation may be obtained by concatenating the respective sub-sequences in the order of increasing length. The inventors of the present invention name the vector sequence X (tails-concatenated) (410) obtained by concatenation as an extended input sequence, and the above-described process for obtaining it as incremental tails-concatenating.

[0058] In Fig. 3, the case where k = 1 is exemplified. At this time, the first sub-sequence (420) is the last vector X L Including only (i.e. {X L}) may have a length of 1, and the second sub-sequence (430) may be L embedded vectors (X1, …, X) of the embedded vector sequence X (embedded) (400). L ) based on the order of the last vector X L and the vector X just before that L-1 Including (i.e., {X L-1 , X L}) may have a length of 2, and the third sub-sequence (440) is L embedded vectors (X1, …, X) of the embedded vector sequence X (embedded) (400). L ) based on the order of the last vector X L , the vector X just before that L-1 , and the vector X just before that L-2 Including (i.e., {X L-2 , X L-1 , X L}) may have a length of 3. Other sub-sequences are generated in the same manner, and the final sub-sequence (460) is L embedded vectors (X1, …, X) of the embedded vector sequence X (embedded) (400). L ) including all of them (i.e. {X1, …, X L} Its length can be L. The order of the vectors in each sub-sequence can be the same as the order in the embedded vector sequence X(embedded)(400). In addition, the vector sequence X(tails-concatenated)(410) obtained by concatenation can be a vector sequence obtained by concatenating each sub-sequence in the order of increasing length. For example, the length of the obtained vector sequence X(tails-concatenated)(410) can be L(L+1) / 2.

[0059] As another example, if the length increasing interval of each sub-sequence is k = 2 (not shown), the first sub-sequence is {X L}(or {X L-1 , X L}), the second sub-sequence is {X L-2 , X L-1 , X L}(or {X L-3 , X L-2 , X L-1 , X L}), the third sub-sequence is {XL-4 , X L-3 , X L-2 , X L-1 , X L}(or {X L-5 , X L-4 , X L-3 , X L-2 , X L-1 , X L}), other sub-sequences are generated in the same manner, and the final sub-sequence is L embedded vectors (X1, ..., X) of the embedded vector sequence X(embedded)(400). L ) can be generated to include all of them. As an example, the length of the first sub-sequence can be set to be k or less, and here, when the length of the first sub-sequence is set to k, the length of the vector sequence X (tails-concatenated) finally obtained by concatenation can be L(L / k+1) / 2. In addition, k can be set among the divisors of L.

[0060] For example, when L = 6 and k = 2, each sub-sequence can be {X6}, {X4, X5, X6}, {X2, X3, X4, X5, X6}, {X1, X2, X3, X4, X5, X6}, or {X5, X6}, {X3, X4, X5, X6}, {X1, X2, X3, X4, X5, X6}. In the latter case, the length of the vector sequence X (tails-concatenated) finally obtained by concatenation becomes 6(6 / 2+1) / 2 = 12.

[0061] In the embodiments described above, an input sequence Input X containing L tokens (or token IDs) initAlthough cases where each sub-sequence is generated from L embedded vectors obtained by first performing embedding on (the first input sequence) have been described, sub-sequences having the same configuration can be obtained even if each sub-sequence is first generated from L tokens (or token IDs) and then embedding is performed. It is obvious to those skilled in the art that obtaining a plurality of sub-sequences through such a modification also falls within the scope of "generating a plurality of sub-sequences based on L vectors each corresponding to the tokens of the first input sequence" in the claims.

[0062] Improved Masking for Followings

[0063] FIG. 5 is a conceptual diagram illustrating a process of an improved masked multi-headed self-attention (300) layer according to one embodiment of the present invention. The masked multi-headed self-attention (300) layer of the embodiment may be included in each encoder / decoder block (e.g., a position corresponding to 130 in FIG. 1) in a transformer (100) structure including n (n is an integer greater than or equal to 1) stacked encoder / decoder blocks (110, 155, 160) as shown in FIG. 1.

[0064] The illustrated improved Masked Multi-Headed Self-Attention (300) layer performs self-attention operations similar to the prior art, and may include h attention heads (350, 360, 370) each operating in parallel. Each attention head (350, 360, 370) obtains a query matrix Q, a key matrix K, and a value matrix V based on an input matrix X (normalized) obtained by performing normalization processing, etc., as needed, based on a predetermined input vector sequence (if the encoder / decoder block is the first block among the stacked encoder / decoder blocks (110, 155, 160), the vector sequence X (tails-concatenated) (410) transmitted from the input transducer (200)) and then multiplies the weight matrices W through matrix multiplication processes (305, 310, 315). Q , W K , W V A query matrix, a key matrix, and a value matrix for processing in each attention head (350, 360, 370) are generated through a process such as performing matrix multiplication operations, etc. Each attention head (350, 360, 370) generates a query-key matrix by performing a matrix multiplication process (320) of the query matrix and the key matrix, and performs a masking process (330) for subsequent tokens.

[0065] Figure 6 is a diagram illustrating an improved subsequent token masking (Masking for Followings) process according to one embodiment of the present invention. Figure 6 illustrates an input sequence Input X init(610) (first input sequence) contains 4 tokens (i.e., L = 4), and the vector sequence X (tails-concatenated) (410) (second input sequence) generated by applying the Incremental Tails-Concatenating technique of the present invention contains 4 sub-sequences, each of which increases in length by 1 (i.e., k = 1). In this case, through the process described in detail above, each sub-sequence can be {X4}, {X3, X4}, {X2, X3, X4}, {X1, X2, X3, X4}. Strictly speaking, the input sequence Input X shown in Fig. 6 init (610) (first input sequence) contains tokens, and vector sequence X (tails-concatenated) (410) (second input sequence) contains vectors. However, for convenience of explanation, specific tokens and their corresponding vectors are represented by the same symbol (e.g., X1, X2, etc.). Table (620) illustrated together in Fig. 6 illustrates each vector in the vector sequence X (tails-concatenated) (410), an index indicating the order or position of these vectors in the vector sequence X (tails-concatenated) (410), and a sub-sequence to which each vector belongs.

[0066] The bottom of Fig. 6 illustrates a case where a general following token masking (Masking for Followings) is applied as is to a query-key matrix (630) generated based on an input vector sequence input to the encoder / decoder block (110, 155, 160) of the stacked encoder / decoder block of Fig. 1 (as described above, if the encoder / decoder block is the first block, it may be a normalized vector sequence X (tails-concatenated) (410) (second input sequence), and if the encoder / decoder block is not the first block, it may be generated through a predetermined process through a lower block). As shown in Fig. 6, the illustrated query-key matrix (630) has rows of i = 1, … , 10 and j = 1, … , 10. , and consists of 10 columns, each row and column corresponding to each input vector in the input vector sequence of the corresponding encoder / decoder block, and also maintaining a correspondence with each vector (see table 620 of Fig. 6) indicated by indices 1, …, 10 in the vector sequence X (tails-concatenated) (410) (second input sequence). The matrix element of the i row and j column of the query-key matrix represents the result of the attention operation (e.g., inner product, etc.) of the i-th input vector and the j-th input vector in the input vector sequence for the corresponding encoder / decoder block.

[0067] In general, the Masking for Followings (330) process is to prevent the embedded vectors of tokens that follow (i.e., appear later) in the input sequence from affecting the self-attention value for the embedded vector of a token, for reasons such as causality, and is thus a well-known technique, so a detailed description thereof will be omitted. In Fig. 6, the matrix elements subjected to masking for following tokens correspond to the area (640) indicated in white in the upper right corner. For masking for following tokens, the values ​​of the matrix elements in the area (640) indicated in white in the upper right corner are replaced with -∞, which means a relatively large negative value, and since it is impossible to process an infinite value on a computer, the result of applying the softmax function is replaced with a relatively large negative value so that it can be ignored.

[0068] Fig. 7 is a diagram for explaining an improved Masking for Followings process according to another embodiment of the present invention. When a general Masking for Followings is applied as is to a query-key matrix (630) generated through a predetermined process based on a vector sequence X (tails-concatenated) (410) to which the Incremental Tails-Concatenating technique of the above-described embodiment is applied, as shown in Fig. 6, a specific vector is generated within the vector sequence X (tails-concatenated) (410) (the second input sequence), but is prior to the actual input sequence Input X init(610) Within the first input sequence, vectors corresponding to tokens that follow (i.e., appear later) the token corresponding to that specific vector can be referenced during the self-attention computation process. The inventors have experimentally recognized that this can lead to malfunction or performance degradation of the transformer, and this is likely due to the influence of causality.

[0069] For example, in Fig. 7, the vector X2 indicated by the index 4 of the vector sequence X (tails-concatenated) (410) (second input sequence) shown in table 620 corresponds to the row of i = 4 and the column of j = 4 of the query-key matrix (730), respectively. However, if the Masking for Followings of the above conventional technology is applied as is in the self-attention operation process, for example, in the case of the 4th row corresponding to the vector X2 indicated by the index 4, the input vectors from the 1st to the 4th column can be referenced, and these input vectors correspond to X4, X3, X4, and X2 on the second input sequence, respectively (see table 620), and among them, X3 and X4 correspond to the actual input sequence Input X init (610) Since the vectors (or tokens) appear after X2, this may cause problems with causality.

[0070] In order to solve these problems, the present embodiment proposes the following new masking process. Here, depending on the implementation example, there may be cases where each token of the input sequence is represented as a row vector rather than a column vector. In such a case, the query-key matrix of FIGS. 6, 7, and 8 has a structure in which the rows and columns are swapped, and the subsequent token masking (330) process of the present embodiment is also applied to match the swapped rows and columns. Therefore, for convenience, in the following, cases in which rows and columns are swapped are expressed in a manner such as column / row or row / column so as to encompass such cases.

[0071] In addition, the area (740, 840) marked in white in the upper right of FIGS. 7 and 8 is an area to which the masking of the known technology described above based on FIG. 6 is applied as is, and therefore will not be described separately.

[0072] The first rule of the masking process of the embodiment

[0073] The first rule of the masking process of the proposed embodiment is that, for a matrix element of row / column i and row / column j of the query-key matrix, if a token of the first input sequence corresponding to the j-th vector of the second input sequence follows a token of the first input sequence corresponding to the i-th vector of the second input sequence in the first input sequence, the corresponding matrix element is replaced with a predetermined value.

[0074] For example, the matrix elements in the 9th row (i = 9) in Fig. 7 correspond to the 9th vector (i.e., index = 9) X3 of the second input sequence, and the matrix elements in the 1st, 3rd, and 6th columns (j = 1, 3, 6) correspond to the 1st, 3rd, and 6th vectors (i.e., index = 1, 3, 6) X4 of the second input sequence, respectively. However, the input sequence Input X init(610) (first input sequence), since the token corresponding to X4 follows the token corresponding to X3, the matrix elements in columns 1, 3, and 6 among the matrix elements in row 9 (i = 9) can be replaced with predetermined values ​​(e.g., relatively large negative values). For the same reason, for example, in the case of the matrix elements in row 9 (i = 9) of FIG. 8, the matrix elements in columns 1, 3, and 6 can be replaced with predetermined values ​​(e.g., relatively large negative values).

[0075] Among the matrix elements of row 9 (i = 9) of FIGS. 7 and 8, the following additional considerations can be made for the remaining matrix elements of columns 2, 4, 5, 7, 8, and 9 after applying masking as above.

[0076] The second rule of the masking process of the embodiment

[0077] The second rule of the masking process of the proposed embodiment is to consider the context within each sub-sequence, that is, for a matrix element of row / column i and row / column j of the query-key matrix, if the set of vectors corresponding to tokens preceding the token corresponding to the j-th vector among the vectors included in the sub-sequence to which the j-th vector of the second input sequence belongs on the first input sequence is not included in the set of vectors corresponding to tokens preceding the token corresponding to the i-th vector among the vectors included in the sub-sequence to which the i-th vector of the second input sequence belongs on the first input sequence, then the corresponding matrix element is replaced with a predetermined value.

[0078] For example, among the matrix elements of row 9 (i = 9) of FIGS. 7 and 8, if we look at the matrix elements of columns 2, 4, 5, 7, 8, and 9 that remain after applying masking as above, row 9 corresponds to X3, the 9th vector in the second input sequence (see Table 620 of FIGS. 7 and 8), and the sub-sequence to which this 9th vector X3 belongs is 4 (i.e.,<X1, X2, X3, X4> ) is. However, among the vectors other than vector X3 included in sub-sequence 4, the set of vectors corresponding to tokens preceding the token corresponding to vector X3 (i.e., tokens corresponding to X1, X2) in the first input sequence (see 610 in Fig. 8) is {X1, X2}. However, the second column of the query-key matrix (730, 830) of FIGS. 7 and 8 corresponds to the second vector X3 in the second input sequence (see table 620 of FIGS. 7 and 8), and the sub-sequence to which this second vector X3 belongs is 2, and among the vectors other than vector X3 included in sub-sequence 2, there are no vectors corresponding to tokens preceding the token corresponding to vector X3 in the first input sequence (see 610 of FIG. 8) (i.e., tokens corresponding to X1, X2), and thus the set of such vectors is the empty set {}, and thus it is included in the set of vectors {X1, X2} described above. In this case, since the input vectors corresponding to the 9th row and the input vectors corresponding to the 2nd column of the query-key matrix (730, 830) in FIGS. 7 and 8 are considered to have no different contexts within each sub-sequence, the matrix elements of the 9th row and 2nd column may not be masked.

[0079] In addition, the input vectors of the 4th column of the query-key matrix (730, 830) of FIGS. 7 and 8 correspond to X2, which is the 4th vector in the 2nd input sequence (see table 620 of FIGS. 7 and 8), and the sub-sequence to which this 4th vector X2 belongs is 3, and among the vectors other than vector X2 included in sub-sequence 3, there is no vector corresponding to a token (i.e., a token corresponding to X1) preceding the token corresponding to vector X2 in the 1st input sequence (see 610 of FIGS. 7 and 8), so the set of such vectors is the empty set {}, and thus it is included in the set of vectors {X1, X2} described above. Therefore, since the input vectors corresponding to the 9th row and the input vectors corresponding to the 4th column of the query-key matrix (730, 830) in FIGS. 7 and 8 are considered to have no different contexts within each sub-sequence, the matrix elements of the 9th row and 4th column may not be masked.

[0080] In addition, the input vector of the 5th column of the query-key matrix (730, 830) of FIGS. 7 and 8 corresponds to X3, which is the 5th vector in the 2nd input sequence (see table 620 of FIGS. 7 and 8), and the sub-sequence to which this 5th vector X3 belongs is 3, and among the vectors other than vector X3 included in sub-sequence 3, the set of vectors corresponding to tokens (i.e., tokens corresponding to X1, X2) preceding the token corresponding to vector X3 in the 1st input sequence (see 610 of FIGS. 7 and 8) is {X2}, so this is included in the set of vectors {X1, X2} described above. Therefore, since the input vectors corresponding to the 9th row and the input vectors corresponding to the 5th column of the query-key matrix (730, 830) in FIGS. 7 and 8 are not considered to have different contexts within each sub-sequence, the matrix elements of the 9th row and 5th column may not be masked.

[0081] In addition, the input vector of the 7th column of the query-key matrix (730, 830) of FIGS. 7 and 8 corresponds to X1, which is the 7th vector in the second input sequence (see table 620 of FIGS. 7 and 8), and the sub-sequence to which this 7th vector X1 belongs is 4, and among the vectors other than vector X1 included in sub-sequence 4, there is no vector corresponding to a token preceding the token corresponding to vector X1 in the first input sequence (see 610 of FIGS. 7 and 8), so the set of such vectors is the empty set {}, and thus it is included in the set {X1, X2} of vectors described above. Therefore, since the input vectors corresponding to the 9th row and the input vectors corresponding to the 7th column of the query-key matrix (730, 830) in FIGS. 7 and 8 are not considered to have different contexts within each sub-sequence, the matrix elements of the 9th row and 7th column may not be masked.

[0082] In addition, the input vector of the 8th column of the query-key matrix (730, 830) of FIGS. 7 and 8 corresponds to X2, which is the 8th vector in the second input sequence (see table 620 of FIGS. 7 and 8), and the sub-sequence to which this 8th vector X2 belongs is 4, and among the vectors other than vector X2 included in sub-sequence 4, the set of vectors corresponding to the token preceding the token corresponding to vector X2 (i.e., the token corresponding to X1) in the first input sequence (see 610 of FIGS. 7 and 8) is {X1}, so this is included in the set of vectors {X1, X2} described above. Therefore, since the input vectors corresponding to the 9th row and the input vectors corresponding to the 8th column of the query-key matrix (730, 830) in FIGS. 7 and 8 are considered to have no different contexts within each sub-sequence, the matrix elements of the 9th row and 8th column may not be masked.

[0083] In addition, the input vector of the 9th column of the query-key matrix (730, 830) of FIGS. 7 and 8 corresponds to X3, which is the 9th vector in the second input sequence (see table 620 of FIGS. 7 and 8), and the sub-sequence to which this 9th vector X3 belongs is 4, and among the vectors other than vector X3 included in sub-sequence 4, the set of vectors corresponding to tokens (i.e., tokens corresponding to X1, X2) preceding the token corresponding to vector X3 in the first input sequence (see 610 of FIGS. 7 and 8) is {X1, X2}, so this is included in the set of vectors {X1, X2} described above. Therefore, since the input vectors corresponding to the 9th row and the input vectors corresponding to the 9th column of the query-key matrix (730, 830) in FIGS. 7 and 8 are not considered to have different contexts within each sub-sequence, the matrix elements of the 9th row and 9th column may not be masked.

[0084] Therefore, when considering the context within each sub-sequence according to the second rule of the masking process of the proposed embodiment, masking may not be applied to the matrix elements in columns 2, 4, 5, 7, 8, and 9 among the matrix elements in row 9 (i = 9) of FIGS. 7 and 8. However, the following additional considerations may be made:

[0085] The third rule of the masking process of the embodiment

[0086] The third rule of the masking process of the proposed embodiment is to exclude distortion of the importance or relevance of each data due to duplicate attention by considering the context based on the processing history of the lower encoder / decoder blocks, etc. In this case, when the matrix elements of multiple columns / rows among the matrix elements of i row / column of the query-key matrix have the same context, in order to avoid duplicate attention, only a predetermined number of the matrix elements of the multiple columns / rows are left, and the rest are replaced with predetermined values.

[0087] As shown in Fig. 1, the i-th output vector of each encoder / decoder block of a neural network (transformer) including n (n is an integer greater than or equal to 1) stacked encoder / decoder blocks (110, 155, 160) may include information of not only the i-th input vector input to the encoder / decoder block but also other input vectors, depending on the result of the self-attention operation performed by each encoder / decoder block. Information of which of the input vectors the i-th output vector of the encoder / decoder block includes may be determined by masking applied in the encoder / decoder block.

[0088] For example, if masking as in FIG. 6 is applied to a certain encoder / decoder block, the output vector corresponding to row 3 (i.e., i = 3) of the encoder / decoder block will also include information on input vectors corresponding to columns 1, 2, and 3 (i.e., j = 1, 2, 3). At this time, since each input vector of the encoder / decoder block in the stacked structure as in FIG. 1 is based on the output vector of the previous encoder / decoder block, these can also selectively include information on the input vectors of the previous encoder / decoder block determined according to the masking applied to the previous encoder / decoder block.

[0089] The inventors of the present invention have experimentally found that the performance of a transformer can be improved by eliminating distortion of the importance or relevance of each data due to redundant attention by considering the context formed according to the processing history of such lower-level encoder / decoder blocks. The inventors define the information regarding which tokens in the first input sequence the ith input vector of a specific encoder / decoder block contains as the 'context information' of the ith input vector of a specific encoder / decoder block.

[0090] Again, in FIGS. 7 and 8, if we look at the matrix elements of the 9th row, 2nd, 4th, 5th, 7th, 8th, and 9th columns of the query-key matrix (730, 830) remaining after applying masking by the first and second rules described above, if the current encoder / decoder block is the first block in the stacked structure of FIG. 1, since there is no previous encoder / decoder block, the input vectors of the current encoder / decoder block are each vector of the vector sequence X(tails-concatenated)(410) (the second input sequence) output from the input transducer (200) of FIG. 2 (i.e., the vectors listed in the table 620 of FIGS. 7 and 8), and therefore, they only contain information about themselves given by the embedding performed in the input transducer (200), and among the input vectors of the previous encoder / decoder block, It does not contain information of other input vectors. That is, the input vectors corresponding to columns 2, 4, 5, 7, 8, and 9 of the query-key matrix (630, 730, 830) are each input sequence Input X init These are vectors corresponding to tokens corresponding to X3, X2, X3, X1, X2, X3 in (the first input sequence), each of which has information only about itself.

[0091] Therefore, when performing the self-attention operation of the current encoder / decoder block, for example, when referencing all input vectors corresponding to columns 2, 4, 5, 7, 8, and 9 for row 9 (columns 1, 3, 6, and 10 are masked by the other masking rules described above), information about X3 is referenced three times and information about X2 is referenced twice, which is redundant. The inventors of the present invention recognized that when information related to specific tokens is referenced redundantly in this way, there is a concern that the importance or relevance between each token to be learned through attention may be reflected in a distorted manner. The third rule of the masking process of the proposed embodiment is to include, in such a case, information about X3 and information about X2 only once each, for example, for row 9, when performing the self-attention operation of the current encoder / decoder block. For example, among the matrix elements of rows 9, columns 2, 4, 5, 7, 8, and 9 of the query-key matrix (730), as shown in FIG. 7, only the matrix elements of columns 7, 8, and 9 can be left, and the rest (i.e., the matrix elements of columns 2, 4, and 5) can be masked. FIG. 7 is an example in which such masking is applied, and as described above, it takes into account the 'context information' of the input vector of the current encoder / decoder block, and can be applied when the current encoder / decoder block is the first block in the stacked structure of FIG. 1.

[0092] Meanwhile, if the current encoder / decoder block is the second or higher block in the stacked structure of FIG. 1, for example, the input vectors corresponding to columns 2, 4, 5, 7, 8, and 9 of the query-key matrix (830) in FIG. 8 are output vectors of the lower encoder / decoder block, so the context information of each input vector may be different from that of the lower encoder / decoder block, and each context information is determined depending on which masking is applied in the lower encoder / decoder block. For example, assuming that the current encoder / decoder block is the second encoder / decoder block and that masking as in FIG. 7 is applied to the first encoder / decoder block, the input vector corresponding to the second column of the query-key matrix (830) of the second encoder / decoder block includes information about X3, the input vector corresponding to the fourth column includes information about X2, the input vector corresponding to the fifth column includes information about X2 and X3, the input vector corresponding to the seventh column includes information about X1, the input vector corresponding to the eighth column includes information about X1 and X2, and the input vector corresponding to the ninth column includes information about X1, X2, and X3.

[0093] Therefore, if the current encoder / decoder block is the second or higher block in the stacked structure of FIG. 1, the input vectors corresponding to columns 2, 4, 5, 7, 8, and 9 of the query-key matrix (830) in FIG. 8 do not have the same context information, and thus cannot be considered to contain the same context information redundantly. In this case, the matrix elements corresponding to rows 9 and columns 2, 4, 5, 7, 8, and 9 of the query-key matrix (830) in FIG. 8 may not be masked. FIG. 8 is an example to which this third masking rule is applied.

[0094] By the embodiments of the present invention described above, it is possible to implement excellent transformer performance through appropriate masking while utilizing attention between sequences of all possible combinations with different starting positions and lengths and tokens included therein for learning as efficiently as possible based on given learning data (corpus).

[0095] In the embodiment of the present invention, the masking technique described with reference to FIG. 7 and the masking technique described with reference to FIG. 8 can be appropriately combined and utilized. For example, in a neural network including n (where n is an integer greater than or equal to 1) stacked encoder or decoder blocks, it is possible to apply the masking technique of FIG. 7 to some encoder or decoder blocks and to apply the masking technique of FIG. 8 to other encoder or decoder blocks. The inventors of the present invention have experimentally recognized that very good transformer performance can be implemented when the masking technique of FIG. 7 is applied to the first encoder or decoder blocks into which the extended vector sequence X (tails-concatenated) (410) is input and the masking technique of FIG. 8 is applied to subsequent encoder or decoder blocks.

[0096] Afterwards, each attention head (350, 360, 370) of the embodiment of FIG. 5 can perform a known SoftMax process (335) on the masked query-key matrix and perform a matrix multiplication operation (340) on the output and the value matrix to generate an output sequence of each attention head. In this way, each of the h attention heads (350, 360, 370) of the masked multi-headed self-attention (300) layer generates its respective outputs through a similar process, and the generated h outputs (Head1, Head2, ..., Head h) are joined together through a concatenation process (380), and the final output of the masked multi-headed self-attention (300) layer is obtained through linear transformation, etc.

[0097] Thereafter, as shown in Fig. 1, the output of the decoder / encoder block is obtained through known layers such as the first Add (135), the second Layer Norm (140), the Pointwise Feed Forward Network (145), and the second Add (150), and can be transferred to another stacked (the number of stacks n is appropriately determined according to the model) encoder / decoder block.

[0098] In the transposed embedding layer (not shown) of the embodiment of the present invention, among the output sequences from the final encoder / decoder block, only the output portion corresponding to the last sub-sequence (or tail sequence) of the vector sequence X (tails-concatenated) (410) (or extended sequence) to which the incremental tails-concatenating technique of the present invention is applied can be converted into a logit through an operation with the transpose matrix of the embedding matrix used in the embedding performed by the input transducer (200). Based on this, probability calculation can be performed in the softmax (170) layer to obtain a probability vector (175) for a token, and a loss (loss, during training) based on cross entropy with the target can be obtained to perform back propagation learning.

[0099] Device to which the proposed method of the present invention can be applied

[0100] Figure 9 illustrates a device (120) capable of performing the proposed method of the present invention.

[0101] Referring to FIG. 9, the device (120) may be configured to implement the proposed method of the present invention and the learning process of a transformer for improving learning efficiency accordingly. For example, the device (120) may be a computing device, a server device, a terminal device, a network device, etc. for performing the process of the present invention.

[0102] For example, the device (120) to which the proposed method of the present invention can be applied may include network devices such as repeaters, hubs, bridges, switches, routers, gateways, etc., computer devices such as desktop computers, workstations, etc., mobile terminals such as smartphones, portable devices such as laptop computers, etc., home appliances such as digital TVs, etc., and transportation means such as automobiles, etc. As another example, the device (120) to which the present invention can be applied may be included as a part of an ASIC (Application Specific Integrated Circuit) implemented in the form of a SoC (System On Chip).

[0103] The memory (20) can be connected to the processor (10) when it operates, and can store programs and / or commands for processing and controlling the processor (10), and can store data and information used in the present invention, control information required for data and information processing according to the present invention, temporary data generated during data and information processing, etc. The memory (20) can be implemented as a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, an SRAM (Static RAM), an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc.

[0104] The processor (10) can be operatively connected to the memory (20) and the network interface (30), and controls the operation of each module within the device (120). In particular, the processor (120) can perform various control functions for performing the proposed method of the present invention. The processor (120) may also be called a controller, a microcontroller, a microprocessor, a microcomputer, etc. The proposed method of the present invention can be implemented by hardware, firmware, software, or a combination thereof. When the present invention is implemented using hardware, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. configured to perform the present invention can be provided in the processor (10). Meanwhile, when implementing the proposed method of the present invention using firmware or software, the firmware or software may include instructions related to modules, procedures, or functions that perform functions or operations necessary to implement the proposed method of the present invention, and the instructions may be stored in the memory (20) or stored in a computer-readable recording medium (not shown) separate from the memory (20) and, when executed by the processor (10), the device (120) may be configured to implement the proposed method of the present invention.

[0105] In addition, the device (120) may include a network interface device (30). The network interface device (30) is connected to the processor (10) when in operation, and the processor (10) may control the network interface device (30) to transmit or receive wireless / wired signals carrying information and / or data, signals, messages, etc. through a wireless / wired network. The network interface device (30) supports various communication standards, such as, for example, IEEE 802 series, 3GPP LTE(-A), 3GPP 5G, etc., and may transmit and receive control information and / or data signals according to the communication standards. The network interface device (30) may also be implemented outside the device (120) as needed.

[0106] The embodiments described above are combinations of components and features of the present invention in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented without being combined with other components or features. Furthermore, it is also possible to form an embodiment of the present invention by combining some components and / or features. The order of operations described in the embodiments of the present invention may be changed. Some components or features of one embodiment may be included in another embodiment or may be replaced with corresponding components or features of another embodiment. It is self-evident that claims that do not have an explicit citation relationship in the patent claims may be combined to form an embodiment or incorporated as a new claim through a post-application amendment.

[0107] The present invention can be applied to various devices such as computing devices, server devices, terminal devices, network devices, etc. for performing learning and inference of attention-based neural networks.

Claims

1. A method performed on a computing device for training a neural network comprising n stacked encoder or decoder blocks (where n is an integer greater than or equal to 1), A step of receiving a first input sequence including L tokens; and A step of generating a plurality of sub-sequences based on L vectors each corresponding to the tokens of the first input sequence, and concatenating the plurality of sub-sequences to generate a second input sequence; and A method comprising the step of performing a self-attention operation based on the second input sequence.

2. In claim 1, A method wherein the plurality of sub-sequences each include at least the last vector among the L vectors.

3. In claim 1, A method wherein the above multiple sub-sequences each have different lengths.

4. In claim 1, A method wherein at least some of the plurality of sub-sequences include vectors each corresponding to a plurality of consecutive tokens on the first input sequence.

5. In claim 1, A method in which the above multiple sub-sequences are generated such that their lengths each increase by k (k is an integer greater than or equal to 1 and less than L).

6. In claim 1, A method wherein the order of vectors included in each of the plurality of sub-sequences is determined based on the order of each token in the first input sequence corresponding to each vector.

7. In claim 1, A method for concatenating the plurality of sub-sequences, wherein the plurality of sub-sequences are concatenated by arranging the lengths of each sub-sequence in increasing order from the shortest to the longest.

8. In claim 1, One or more of the encoder or decoder blocks generates a query-key matrix to perform the self-attention operation, and performs a subsequent token masking process on the query-key matrix, The above subsequent token masking process is: A method comprising replacing a matrix element of the i row / column and j column / row of the query-key matrix with a predetermined value, if a token of the first input sequence corresponding to the j-th vector of the second input sequence follows a token of the first input sequence corresponding to the i-th vector of the second input sequence in the first input sequence.

9. In claim 1, One or more of the encoder or decoder blocks generates a query-key matrix to perform the self-attention operation, and performs a subsequent token masking process on the query-key matrix, The above subsequent token masking process is: A method comprising replacing the matrix element of the i row / column and j column / row of the query-key matrix with a predetermined value, if a set of vectors corresponding to tokens preceding the token corresponding to the j-th vector in the first input sequence among vectors included in the sub-sequence to which the j-th vector of the second input sequence belongs is not included in the set of vectors corresponding to tokens preceding the token corresponding to the i-th vector in the first input sequence among vectors included in the sub-sequence to which the i-th vector of the second input sequence belongs.

10. In claim 1, One or more of the encoder or decoder blocks generates a query-key matrix to perform the self-attention operation, and performs a subsequent token masking process on the query-key matrix, The above subsequent token masking process is: A method comprising, in order to avoid duplicate attention, leaving only a predetermined number of matrix elements of the plurality of columns / rows among the matrix elements of the i row / column of the query-key matrix and replacing the rest with predetermined values, when the matrix elements of multiple columns / rows have the same context.

11. In claim 1, A method in which the neural network calculates a loss by using only the output corresponding to the last sub-sequence among the plurality of sub-sequences of the second input sequence among the output sequences processed through the stacked encoder or decoder blocks of the second input sequence.

12. In a device including a processor and memory, The memory comprises instructions configured to cause the device, when executed by the processor, to implement a specific operation for training a neural network comprising n stacked encoder or decoder blocks, where n is an integer greater than or equal to 1, the specific operation comprising: Receiving a first input sequence containing L tokens; and Generating a plurality of sub-sequences based on L vectors each corresponding to the tokens of the first input sequence, and concatenating the plurality of sub-sequences to generate a second input sequence; and A device comprising performing a self-attention operation based on the second input sequence.

13. A computer-readable non-transitory storage medium storing instructions configured to cause a device including said processor to implement a specific operation for training a neural network including n stacked encoder or decoder blocks, wherein n is an integer greater than or equal to 1, wherein said specific operation comprises: Receiving a first input sequence containing L tokens; and Generating a plurality of sub-sequences based on L vectors each corresponding to the tokens of the first input sequence, and concatenating the plurality of sub-sequences to generate a second input sequence; and A computer-readable non-transitory storage medium comprising performing a self-attention operation based on the second input sequence.

Images