Information processing device and program

Abstract

To provide an information processing device (transformer model) and a program therefor that improve the way of handling positional encoding in order to improve processing performance.SOLUTION: An information processing device includes an encoder unit that obtains intermediate data based on input data, and a decoder unit that obtains data for output based on the intermediate data. An encoder-side multi-head attention layer inputs the input data that has been subjected to positional encoding as a query (Q) and a key (K), and inputs the input data that has not been subjected to the positional encoding as a value (V). A decoder-side multi-head attention layer inputs the output data for output that has been subjected to the positional encoding as a query (Q) and a key (K), and inputs the output data for output that has not been subjected to the positional encoding as a value (V).SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device and a program. [Background technology]

[0002] Transformer models have become a widely used information processing method in recent years. Transformer models are often used in natural language processing, including machine translation, but their applications are not limited to natural language processing. When using a Transformer model for natural language translation, information about word order within a sentence is important. In a Transformer model, word order information is represented as a vector called Positional Encoding (PE).

[0003] In other words, information about word order (position within a sentence) is embedded in the input to the Transformer model. There are two main methods for embedding positional information. The first is absolute positional encoding (Absolute PE), and the second is relative positional encoding (Relative PE). In absolute positional encoding, a vector representing the absolute position of each word (1, 2, 3, ...) is added to the vector of the word, regardless of the word of interest. In relative positional encoding, a vector representing the relative distance from the word of interest (..., -2, -1, 0, 1, 2, ...) is added to the vector of the word. When comparing these two types of positional information embedding, relative positional encoding is said to perform better. Relative positional encoding is said to perform better, especially when the input to the Transformer model is a long sentence. However, relative positional encoding requires more computational effort, so absolute positional encoding is often used.

[0004] A transformer model is proposed in Non-Patent Document 1. Positional encoding is also proposed in Non-Patent Document 1.

[0005] Non-Patent Document 2 analyzes translation performance when translating longer sentences that do not appear in training data, and also reports that Relative PE has good performance.

[0006] 7 is a functional block diagram showing the general functional configuration of a Transformer model (information processing device 99) according to conventional technology. The functional configuration of this Transformer model itself is already well known. As shown in the figure, the Transformer model (information processing device 99) includes an encoder unit 10 and a decoder unit 60. The Transformer model (information processing device 99) also includes an input encoding unit 11, an adder unit 12, an output encoding unit 61, and an adder unit 62.

[0007] As shown in the figure, in the conventional Transformer model, an adder 12 adds positional encoding (PE) to the output from an input encoding unit 11, and the result is input to an encoder unit 10. An adder 62 adds positional encoding (PE) to the output from an output encoding unit 61, and the result is input to a decoder unit 60. In other words, the output value from the adder 12 is the value of all inputs V (value), K (key), and Q (query) of a multi-head attention unit 13. Similarly, the output value from the adder 62 is the value of all inputs V (value), K (key), and Q (query) of a masked multi-head attention unit 63. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, Attention Is All You Need, arXiv:1706.03762, 2017. [Non-patent document 2] Jan Rosendahl,Viet Anh Khoa Tran,Weiyue Wang,Hermann Ney,Analysis of Positional Encodings for Neural Machine Translation,In Proceedings of the 16th International Conference on Spoken Language Translation,Hong Kong,Association for Computational Linguistics,2019. Summary of the Invention [Problem to be solved by the invention]

[0009] In the conventional Transformer model, positional encoding is applied to all three inputs to the multi-head attention unit. As mentioned above, the three inputs are V (value), K (key), and Q (query). However, this conventional method has the problem that the multi-head attention unit outputs more vectors derived from positional encoding. The vectors output from the multi-head attention unit are ultimately used to determine which word to output, so in reality, so many vectors derived from positional encoding should not be necessary.

[0010] The present invention was made based on the above-mentioned problem recognition and aims to improve the handling of positional encoding within the Transformer model. In other words, the present invention aims to provide an information processing device (Transformer model) and a program therefor that improve the way positional encoding is handled in order to improve processing performance (translation performance in the case of translation). [Means for solving the problem]

[0011] [1] In order to solve the above-mentioned problems, an information processing device according to one aspect of the present invention is an information processing device comprising: an encoder unit that obtains intermediate data based on input data; a first positional encoding application unit that applies positional encoding to the input data; a decoder unit that obtains output data based on the intermediate data; and a second positional encoding application unit that applies positional encoding to output data output by the decoder unit, wherein the encoder unit receives the input data that has been subjected to positional encoding as a query (Q) and a key (K), and receives the input data that has not been subjected to positional encoding as a value (V), and includes an encoder-side multi-head attention layer that obtains data of a weighted sum of the value (V) based on the similarity between the query (Q) and the key (K); and a feedforward network that generates a weighted sum of the value (V) based on the data output from the encoder-side multi-head attention layer. the decoder unit comprises a decoder-side multi-head attention layer that inputs the output data that has been subjected to positional encoding as a query (Q) and a key (K), inputs the output data that has not been subjected to positional encoding as a value (V), and calculates data of the weighted sum of the value (V) based on the similarity between the query (Q) and the key (K); a cross-attention layer that inputs the intermediate data output from the encoder-side feedforward layer as a value (V) and a key (K), inputs data output from the decoder-side multi-head attention layer as a query (Q), and calculates data of the weighted sum of the value (V) based on the similarity between the query (Q) and the key (K); and an encoder-side feedforward layer that processes the feedforward network based on the data output from the cross-attention layer.

[0012] [2] Also, one aspect of the present invention is that in the information processing device of [1] above, the decoder unit further includes a third positional encoding action unit that applies positional encoding to the intermediate data output from the encoder-side feedforward layer, and a fourth positional encoding action unit that applies positional encoding to the data output from the decoder-side multi-head attention layer, and the cross-attention layer inputs the intermediate data that has not been subjected to positional encoding as a value (V), inputs the intermediate data that has been subjected to positional encoding as a key (K), and inputs the data that is output from the decoder-side multi-head attention layer and that has been subjected to positional encoding as a query (Q).

[0013] [3] Furthermore, one aspect of the present invention is an information processing device according to the above [1] or [2], configured to update internal parameters by performing error backpropagation based on the difference between estimated data, which is output data estimated based on the input data included in the learning data, and the correct answer data included in the learning data, based on learning data which is a pair of input data and correct answer data which is the correct answer to output data.

[0014] [4] Another aspect of the present invention is that, in any of the information processing devices described above in [1] to [3], the input data is data of a token sequence of a source language sentence, and the output data is data of a token sequence of a target language sentence corresponding to the source language sentence.

[0015] [5] Also, one aspect of the present invention is a program that causes a computer to function as an information processing device comprising: an encoder unit that obtains intermediate data based on input data; a first positional encoding application unit that applies positional encoding to the input data; a decoder unit that obtains output data based on the intermediate data; and a second positional encoding application unit that applies positional encoding to output data output by the decoder unit, wherein the encoder unit inputs the input data that has been subjected to positional encoding as a query (Q) and a key (K), and inputs the input data that has not been subjected to positional encoding as a value (V), and includes an encoder-side multi-head attention layer that obtains data of a weighted sum of the value (V) based on the similarity between the query (Q) and the key (K); and a feedforward network processing unit that performs processing based on the data output from the encoder-side multi-head attention layer. the decoder unit receives as input the output data that has been subjected to positional encoding as a query (Q) and a key (K), receives as input the output data that has not been subjected to positional encoding as a value (V), and calculates data of a weighted sum of the value (V) based on the similarity between the query (Q) and the key (K); a cross-attention layer receives as input the intermediate data output from the encoder-side feedforward layer as a value (V) and a key (K), receives as input the data output from the decoder-side multi-head attention layer as a query (Q), and calculates data of a weighted sum of the value (V) based on the similarity between the query (Q) and the key (K); and an encoder-side feedforward layer that processes a feedforward network based on the data output from the cross-attention layer. [Effects of the Invention]

[0016] According to the present invention, neither the output from the encoder-side multi-head attention layer nor the output from the decoder-side multi-head attention layer contains more vectors derived from positional encoding. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram showing a schematic functional configuration of an information processing device (Transformer model) according to a first embodiment of the present invention. [Figure 2] FIG. 3 is a block diagram showing another schematic functional configuration of the information processing device (Transformer model) according to the first embodiment. [Figure 3] FIG. 10 is a block diagram showing a schematic functional configuration of an information processing device (Transformer model) according to a second embodiment. [Figure 4] FIG. 10 is a block diagram showing another schematic functional configuration of the information processing device (Transformer model) according to the second embodiment. [Figure 5] FIG. 2 is a block diagram showing an example of an internal configuration for realizing an information processing device in the first and second embodiments. [Figure 6] 1 is a graph showing translation performance, which is a result of demonstration experiments of the first and second embodiments, as a function of sentence length (token length). [Figure 7] FIG. 1 is a functional block diagram showing a schematic functional configuration of a transformer model (information processing device) according to a conventional technique. DETAILED DESCRIPTION OF THE INVENTION

[0018] Next, several embodiments of the present invention will be described with reference to the drawings. These embodiments are improvements on the Transformer model.

[0019] [First embodiment] 1 is a block diagram showing a schematic functional configuration of an information processing device (Transformer model) according to a first embodiment of the present invention. As shown in the figure, the information processing device 100 includes an input encoding unit 11, an adder unit 112, an encoder unit 110, an output encoding unit 61, an adder unit 162, a decoder unit 160, a linear transformation processing unit 69, and a softmax function processing unit 70.

[0020] Each function of the information processing device 100 can be realized, for example, by a computer and a program. Each function uses a storage means as needed. The storage means is, for example, a variable in the program or a memory allocated by the execution of the program. Non-volatile storage means such as a magnetic hard disk drive or a solid-state drive (SSD) may also be used as needed. At least a part of each function may also be realized as a dedicated electronic circuit rather than a program.

[0021] The information processing device 100 is implemented using a neural network. In other words, the values ​​of the parameters internal to the information processing device 100 are optimized by machine learning. During training, the information processing device 100 performs training using training data, and updates the internal parameters using backpropagation. The updated parameter values ​​are stored in a storage unit. In other words, the information processing device 100 is configured to update the internal parameters by performing backpropagation based on the difference between estimated data, which is output data estimated based on the input data included in the training data, and the correct data included in the training data, based on training data that is a pair of input data and correct answer data that is the correct answer for output data. When performing estimation processing (estimating output data for various types of tasks) using the trained information processing device 100, forward propagation processing is performed using the optimized parameter values.

[0022] The encoder unit 110 performs processing to obtain intermediate data based on the input data, and passes the calculated intermediate data to the decoder unit 160.

[0023] The decoder unit 160 performs processing to obtain output data based on the intermediate data passed from the encoder unit 110 .

[0024] When the information processing device 100 performs a process of translating a natural language sentence, the input data is data on a token sequence of a source language sentence. The output data is data on a token sequence of a target language sentence corresponding to the source language sentence. In other words, the information processing device 100 performs a process of determining (estimating) data on a token sequence of a target language sentence based on data on a token sequence of a source language sentence. Note that the task performed by the information processing device 100 is not limited to translating natural language sentences, and various other tasks can be performed.

[0025] In the figure, the encoder unit 110 and the decoder unit 160 are each denoted as "Nx" because each of them is made up of N layers (N is a positive integer). N is a positive integer determined appropriately, and may be, for example, N=6. In other words, the encoder unit 110 repeats the same process N times. The decoder unit 160 also repeats the same process N times. A more detailed internal functional configuration of each of the encoder unit 110 and the decoder unit 160 is as follows.

[0026] As shown in the figure, the encoder unit 110 includes a multi-head attention unit 13, an addition and normalization processing unit 14, a feedforward processing unit 15, an addition and normalization processing unit 16, and an addition unit 112. The decoder unit 160 includes a masked multi-head attention unit 63, an addition and normalization processing unit 64, a multi-head attention unit 65, an addition and normalization processing unit 66, a feedforward processing unit 67, an addition and normalization processing unit 68, and an addition unit 162.

[0027] Detailed functions of each unit constituting the information processing device 100 are as follows: First, the multi-head attention processing, the feedforward network processing, and the positional encoding will be explained, and then the overall processing flow of the information processing device 100 will be explained.

[0028] [Handling multi-head attention] Multi-Head Attention processing uses the attention mechanism to perform calculations. Specifically, Multi-Head Attention processing inputs V (value), K (key), and Q (query), and calculates the output value using the following formula (1).

[0029]

number

[0030] In equation (1), Q, K, and V are data (vectors) input to the multi-head attention process. T is the inner product of Q and K, and represents the similarity between them. The function Softmax() performs calculations to make the sum of similarities 1 (normalization). The denominator √D in the argument of the function Softmax() is given to prevent gradient vanishing. Note that D is the number of dimensions of the vectors Q, K, and V. In other words, the Output (output value) in equation (6) is the sum of V weighted by the similarity.

[0031] [Feedforward network processing] The processing of a feedforward network involves the forward propagation of a neural network. Specifically, the processing of a feedforward network consists of two linear transformation processes sandwiched between the processing of a ReLU (Rectified Linear Unit) function. The ReLU function is a function that outputs 0 if the input value is less than or equal to 0, and outputs the input value unchanged if the input value is positive.

[0032] [Positional Encoding] Positional encoding is information that represents the order of target data. When the target data is sentence data (token sequence data corresponding to a sentence) that is the target of machine translation, positional encoding is information that represents the order of tokens contained in the sequence. By applying (e.g., adding) the positional encoding value to the token, order (position) information is embedded in the token data. The positional encoding value may be, for example, as shown in equations (2) and (3) below.

[0033]

number

[0034]

number

[0035] In equations (2) and (3), pos is the position of the token, i is the index value of each dimension in the vector corresponding to the token, and d model is the number of dimensions of the vector corresponding to the token.

[0036] The input encoding unit 11 encodes (compresses) the input data. The output (vector) from the input encoding unit 11 is passed to the adder 112, the multi-head attention unit 13 of the encoder unit 110, and the adder and normalization unit 14.

[0037] The adder 112 adds a positional encoding (PE) vector to the vector passed from the input encoding unit 11. The positional encoding vector may be based on relative positional encoding or absolute positional encoding. The output from the adder 112 becomes the input of K (key) and Q (query) of the multi-head attention unit 13. However, the output from the adder 112 does not become the input to V (value) of the multi-head attention unit 13.

[0038] The inputs to the encoder unit 110 are as follows: the output from the adder unit 112 (to which positional encoding has been added) is input to K and Q of the multi-head attention unit 13. The output from the input encoding unit 11, which does not pass through the adder unit 112, is input to V of the multi-head attention unit 13 and is also passed to the addition and normalization processing unit 14.

[0039] In each layer within the encoder unit 110 (each layer included in the N layers), the following processing is performed by a multi-head attention unit 13, an addition and normalization processing unit 14, a feedforward processing unit 15, and an addition and normalization processing unit 16.

[0040] The multi-head attention unit 13 performs attention mechanism processing (multi-head attention processing) based on the input vector. That is, the multi-head attention unit 13 performs processing to obtain the weighted sum data of V based on the similarity between the Q input and the K input. As mentioned above, positional encoding is not added to the V input of the multi-head attention unit 13, and the input data becomes the input value to V as is. Positional encoding is added to each of the K and Q input values ​​of the multi-head attention unit 13 by the adder 112. The output from the multi-head attention unit 13 is passed to the adder and normalization processor 14.

[0041] The addition and normalization processing unit 14 performs residual connection and normalization processing (Add&Norm layer processing) based on the output from the multi-head attention unit 13 and the original data. The output from the addition and normalization processing unit 14 is passed to the feedforward processing unit 15 and the addition and normalization processing unit 16.

[0042] The feedforward processing unit 15 performs processing using a feedforward network based on the vector passed from the addition and normalization processing unit 14. The output from the feedforward processing unit 15 is passed to the addition and normalization processing unit 16.

[0043] The addition and normalization processing unit 16 performs residual connection and normalization processing (processing of the Add&Norm layer) based on the output from the feedforward processing unit 15 and the output from the preceding addition and normalization processing unit 14. The output from the addition and normalization processing unit 16 becomes the output from the encoder unit 110.

[0044] The output from the final layer of the encoder unit 110 is passed to the multi-head attention unit 65 of the decoder unit 160. The information passed from the encoder unit 110 to the decoder unit 160 is a vector that represents the content of each token with context added within the encoder.

[0045] The configuration of the decoder unit 160 is partially common to the configuration of the encoder unit 110 described above. That is, the multi-head attention unit 13 of the encoder unit 110 corresponds to the masked multi-head attention unit 63 of the decoder unit 160. The addition and normalization processing unit 14 of the encoder unit 110 corresponds to the addition and normalization processing unit 64 of the decoder unit 160. The feedforward processing unit 15 of the encoder unit 110 corresponds to the feedforward processing unit 67 of the decoder unit 160. The addition and normalization processing unit 16 of the encoder unit 110 corresponds to the addition and normalization processing unit 68 of the decoder unit 160. In addition to the configuration of the masked multi-head attention unit 63, normalization processing unit 64, feedforward processing unit 67, and addition and normalization processing unit 68, the decoder unit 160 also includes a multi-head attention unit 65 that receives the output from the addition and normalization processing unit 64, and an addition and normalization processing unit 66 that processes the output of the multi-head attention unit 65. The processing on the decoder unit 160 side is as follows.

[0046] The output encoding unit 61 shifts and encodes (compresses) the output data. The output from the output encoding unit 61 is passed to the adder 162, the masked multi-head attention unit 63 of the decoder unit 160, and the adder and normalizer 64.

[0047] The adder 162 adds a positional encoding (PE) vector to the vector passed from the output encoding unit 61. The output from the adder 162 becomes the input of K (key) and Q (query) of the masked multi-head attention unit 63. However, the output from the adder 162 does not become the input of V (value) of the masked multi-head attention unit 63.

[0048] The inputs to the decoder unit 160 are as follows: the output from the adder unit 162 (to which positional encoding has been added) is input to K and Q of the masked multi-head attention unit 63. The output from the output encoding unit 61, which does not pass through the adder unit 162, is input to V of the masked multi-head attention unit 63 and is also passed to the addition and normalization processing unit 64.

[0049] The masked multi-head attention unit 63 performs attention mechanism processing based on the input vector. Note that positional encoding is not added to the V input of the masked multi-head attention unit 63, and the input data becomes the input value to V as is. Positional encoding is added to the input values ​​of K and Q of the masked multi-head attention unit 63 by the adder 162. The output from the masked multi-head attention unit 63 is passed to the adder and normalization processor 64.

[0050] The masked multi-head attention unit 63 has a function of masking information. In other words, the masked multi-head attention unit 63 masks input information by, for example, replacing part of the input data with a fixed value. Note that when training a neural network for natural language translation processing, this mask is used to train using input under the same conditions as during decoding without referring to untranslated future information, but the mask is not used during translation.

[0051] The addition and normalization processing unit 64 performs residual connection and normalization processing (Add&Norm layer processing) based on the output from the masked multi-head attention unit 63 and the data from the original output encoding unit 61. The output from the addition and normalization processing unit 64 is passed to the multi-head attention unit 65 and the addition and normalization processing unit 66.

[0052] The multi-head attention unit 65 performs processing of the attention mechanism. The V and K inputs of the multi-head attention unit 65 are the output values ​​from the addition and normalization processing unit 16 of the encoder unit 110 (intermediate data passed from the encoder unit 110 to the decoder unit 160). The Q input of the multi-head attention unit 65 is the output value from the addition and normalization processing unit 64. In other words, the cross-attention layer 565 performs processing to obtain the weighted sum data of V based on the similarity between the above Q and the above K.

[0053] The addition and normalization processing unit 66 performs residual connection and normalization processing (Add&Norm layer processing) based on the output from the multi-head attention unit 65 and the original data from the addition and normalization processing unit 64. The output from the addition and normalization processing unit 66 is passed to the feedforward processing unit 67.

[0054] The feedforward processing unit 67 performs processing using a feedforward network based on the vector passed from the addition and normalization processing unit 66. The output from the feedforward processing unit 67 is passed to the addition and normalization processing unit 68.

[0055] The addition and normalization processing unit 68 performs residual connection and normalization processing (Add&Norm layer processing) based on the output from the feedforward processing unit 67 and the output from the preceding addition and normalization processing unit 66. The output from the addition and normalization processing unit 68 becomes the output from the decoder unit 160.

[0056] The linear transformation processing unit 69 performs linear transformation processing based on the data passed from the addition and normalization processing unit 68 (output from the decoder unit 160). The linear transformation processing unit 69 is implemented using a neural network. The values ​​of the weights used when the linear transformation processing unit 69 performs linear transformation are internal parameters. In other words, the values ​​of the internal parameters held by the linear transformation processing unit 69 can also be updated through learning.

[0057] The softmax function processing unit 70 performs softmax function processing based on the output from the linear transformation processing unit 69. The softmax function processing unit 70 normalizes each of the values ​​output from the linear transformation processing unit 69.

[0058] As described above, the features of this embodiment are as follows: The data (input data) to which the adder unit 112 has added positional encoding becomes the inputs K and Q of the multi-head attention unit 13. The data (input data) to which the adder unit 112 has not added positional encoding becomes the input V of the multi-head attention unit 13. Furthermore, the data to which the adder unit 162 has added positional encoding (shifted output data) becomes the inputs K and Q of the masked multi-head attention unit 63. The data to which the adder unit 162 has not added positional encoding (shifted output data) becomes the input V of the masked multi-head attention unit 63. Other processing is basically the same as that of the Transformer model according to existing technology.

[0059] Fig. 2 is a block diagram showing the schematic functional configuration of the information processing device (Transformer model) according to the first embodiment in a form different from that shown in Fig. 1. As shown in the figure, the information processing device 100 includes an input encoding unit 11, a first positional encoding application unit 612, an encoder unit 110, an output encoding unit 61, a second positional encoding application unit 662, a decoder unit 160, a linear transformation processing unit 69, and a softmax function processing unit 70.

[0060] The first positional encoding application unit 612 corresponds to the addition unit 112 in Fig. 1. The first positional encoding application unit 612 is also called a "first PE application unit" for short. The first positional encoding application unit 612 performs processing to apply positional encoding to input data.

[0061] The second positional encoding application unit 662 corresponds to the addition unit 162 in Fig. 1. The second positional encoding application unit 662 is also referred to as the "second PE application unit" for short. The second positional encoding application unit 662 performs a process of applying positional encoding to the output data (however, right shifted) output by the decoder unit 160.

[0062] 2, the encoder unit 110 includes an encoder-side multi-head attention layer 513 and an encoder-side feedforward layer 515. The decoder unit 160 includes a decoder-side multi-head attention layer 563, a cross-attention layer 565, and a decoder-side feedforward layer 567.

[0063] The encoder-side multi-head attention layer 513 corresponds to the multi-head attention unit 13 and the addition and normalization processing unit 14 in Fig. 1. In other words, the encoder-side multi-head attention layer 513 receives input data that has been subjected to positional encoding as a query (Q) and a key (K), receives input data that has not been subjected to positional encoding as a value (V), and performs processing to obtain the weighted sum data of the value (V) based on the similarity between the query (Q) and the key (K).

[0064] The encoder-side feedforward layer 515 corresponds to the feedforward processing unit 15 and the addition and normalization processing unit 16 in Fig. 1. In other words, the encoder-side feedforward layer 515 performs feedforward network processing based on the data output from the encoder-side multi-head attention layer 513.

[0065] The decoder-side multi-head attention layer 563 corresponds to the masked multi-head attention unit 63 and the addition and normalization processing unit 64 in Fig. 1. In other words, the decoder-side multi-head attention layer 563 receives the output data (right shifted) that has been subjected to positional encoding as the query (Q) and key (K), receives the output data that has not been subjected to positional encoding as the value (V), and performs processing to obtain the weighted sum data of the value (V) based on the similarity between the query (Q) and the key (K).

[0066] The cross-attention layer 565 corresponds to the multi-head attention unit 65 and the addition and normalization processing unit 66 in Fig. 1. In other words, the cross-attention layer 565 receives the intermediate data output from the encoder-side feedforward layer 515 as values ​​(V) and keys (K), receives the data output from the decoder-side multi-head attention layer 563 as queries (Q), and performs processing to obtain the weighted sum data of the values ​​(V) based on the similarity between the query (Q) and the key (K).

[0067] The decoder-side feedforward layer 567 corresponds to the feedforward processing unit 67 and the addition and normalization processing unit 68 in Figure 1. In other words, the decoder-side feedforward layer 567 performs feedforward network processing based on the data output from the cross-attention layer 565.

[0068] As explained above, the Transformer's multi-head attention (MHA) processing involves inputs of value (V), key (K), and query (Q). In this embodiment, rather than adding positional encoding (PW) to all of these value (V), key (K), and query (Q) inputs, positional encoding is added only to the key (K) and query (Q) inputs. In other words, positional encoding is not added to the value (V) input. These improvements aim to improve performance.

[0069] This improvement will improve translation performance, especially generalization performance for long texts (ultra-short sentences), such as breaking news, for which it is difficult to collect training data.

[0070] [Second embodiment] Next, a second embodiment of the present invention will be described. Note that the following description may omit matters that have already been described in the previous embodiment. Here, the description will focus on matters unique to this embodiment.

[0071] 3 is a block diagram showing a schematic functional configuration of an information processing device (Transformer model) according to a second embodiment of the present invention. As shown in the figure, the information processing device 200 includes an input encoding unit 11, an adder unit 112, an encoder unit 210, an output encoding unit 61, an adder unit 162, a decoder unit 260, a linear transformation processing unit 69, and a softmax function processing unit 70.

[0072] A feature of this embodiment is that the decoder unit 260 includes an adder unit 271 and an adder unit 272 at the input locations of K and Q to the multi-head attention unit 65, respectively. The adder unit 271 adds positional encoding to the value (vector) input as K to the multi-head attention unit 65. The adder unit 272 also adds positional encoding to the value (vector) input as Q to the multi-head attention unit 65. Note that positional encoding is not added for the input of V to the multi-head attention unit 65. Except for this feature, the configuration and functions of the information processing device 200 in this embodiment are similar to those of the information processing device 100 in the first embodiment.

[0073] That is, in the information processing device 200 of this embodiment, positional encoding is also applied to each of the inputs of K and Q to the multi-head attention unit 65.

[0074] Fig. 4 is a block diagram showing the schematic functional configuration of an information processing device (Transformer model) according to the second embodiment in a form different from that of Fig. 3. As shown in the figure, the information processing device 200 includes an input encoding unit 11, a first positional encoding application unit 612, an encoder unit 210, an output encoding unit 61, a second positional encoding application unit 662, a decoder unit 260, a linear transformation processing unit 69, and a softmax function processing unit 70.

[0075] The first positional encoding application unit 612 and the second positional encoding application unit 662 are as described with reference to Fig. 2. That is, the first positional encoding application unit 612 performs a process of applying positional encoding to input data. Also, the second positional encoding application unit 662 performs a process of applying positional encoding to the output data (however, shifted to the right) that has been output by the decoder unit 160.

[0076] 4, the encoder unit 210 includes an encoder-side multi-head attention layer 513 and an encoder-side feedforward layer 515. The decoder unit 260 includes a decoder-side multi-head attention layer 563, a cross-attention layer 565, a decoder-side feedforward layer 567, a third positional encoding effect unit 771, and a fourth positional encoding effect unit 772.

[0077] The configuration and functions of the encoder unit 210 are similar to those of the encoder unit 110 described with reference to FIG.

[0078] Of the components of the decoder unit 260, the decoder-side multi-head attention layer 563, the cross-attention layer 565, and the decoder-side feedforward layer 567 are as described with reference to FIG.

[0079] However, in Figure 4, among the V, K, and Q inputs of the cross-attention layer 565, the K and Q inputs have been subjected to the action of positional encoding. That is, as follows:

[0080] The third positional encoding application unit 771 corresponds to the addition unit 271 in Fig. 3. The third positional encoding application unit 771 is also referred to as the "third PE application unit" for short. In other words, the third positional encoding application unit 771 performs processing to apply positional encoding to the intermediate data output from the encoder-side feedforward layer 515.

[0081] The fourth positional encoding application unit 772 corresponds to the addition unit 272 in Fig. 3. The fourth positional encoding application unit 772 is also referred to as the "fourth PE application unit" for short. That is, the fourth positional encoding application unit 772 applies positional encoding to the data output from the decoder-side multi-head attention layer 563.

[0082] With the above configuration, the cross-attention layer 565 inputs the intermediate data that has not been subjected to positional encoding as a value (V), inputs the intermediate data that has been subjected to positional encoding as a key (K), and inputs the data output from the decoder-side multi-head attention layer 563 that has been subjected to positional encoding as a query (Q).

[0083] In addition to the effects of the first embodiment, this embodiment also produces the following effect: In this embodiment, positional encoding is added to the inputs of the key (K) and query (Q) in the cross-attention layer, thereby further improving performance.

[0084] FIG. 5 is a block diagram showing an example of the internal configuration for realizing an information processing device in the first or second embodiment. At least some of the functions of the information processing device 100 or 200 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices, etc. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port 903 via the bus 906.

[0085] At least some of the functions of the information processing device in the first and second embodiments described above can be implemented by a computer and a program. In this case, the functions can be implemented by recording a program for implementing the functions on a computer-readable recording medium and loading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases. The program may also be designed to implement some of the functions described above, or may be capable of implementing the functions described above in combination with a program already stored in the computer system.

[0086] [Variations of the embodiment] The first and second embodiments have been described above, but variations of these embodiments are as follows. It has already been explained that each of the encoder units (110, 210) and decoder units (160, 260) may be configured with N layers (N is a positive integer). As a variation when N≧2, positional encoding may be applied only to the first layer (first layer), or may be applied to the second layer and subsequent layers.

[0087] Although several embodiments of the present invention have been described in detail above with reference to the drawings, the specific configurations are not limited to these embodiments, and designs within the scope of the present invention are also included.

[0088] The information processing device according to the first or second embodiment can also be used to perform processes other than natural language translation (translation from a source language sentence to a target language sentence). The information processing device can also perform processes such as analyzing or classifying text written in a natural language, classifying emotions expressed in text written in a natural language, creating a summary of text written in a natural language, creating an answer to a question written in a natural language, generating sentences (including automatic writing of documents, articles, etc.), generating code for a computer program, generating images, generating music, interpreting information, and other processes.

[0089] [Evaluation experiment] Next, we will explain the results of the evaluation experiment. In this experiment, we evaluated the accuracy of natural language translation. The experimental conditions are as follows: The data used in the experiment was FR-DE (translation data from French to German) and JA-EN (translation data from Japanese to English) from WMT22 (Conference on Machine Translation 2022). The amount (number) of data used for training data, development data, and evaluation data for FR-DE and JA-EN, respectively, is shown in Table 1 below.

[0090] [Table 1]

[0091] The results of the French to German translation are shown in Table 2 below. In the experiment, absolute positional encoding was used to evaluate and compare the prior art (FIG. 7), the first embodiment of this specification (FIG. 1), and the second embodiment of this specification (FIG. 3). Relative positional encoding was also used to evaluate and compare the first embodiment of this specification (FIG. 1) and the second embodiment of this specification (FIG. 3).

[0092] As already explained, in the prior art, the result of adding positional encoding is input to all of V, K, and Q of the multi-head attention unit. In the first embodiment (FIG. 1), the result of adding positional encoding is input only to K and Q of the multi-head attention units (13 and 63), not to V. The technique used in the first embodiment is referred to herein as "less-feeding" (supplying less positional encoding). In the second embodiment (FIG. 3), the above-mentioned less-feeding technique is used, and positional encoding is also added to each of the K and Q inputs of the multi-head attention unit 65 located in the middle layer of the decoder unit 260. This technique used in the second embodiment is referred to herein as "cross-PE" (cross-positional encoding). In other words, the second embodiment uses both the less-feeding and cross-PE techniques. Note that BLEU and COMET are used as evaluation scores.

[0093] [Table 2]

[0094] In French-to-German translations, both the first and second embodiments achieved higher scores than the conventional technology (which does not use either the less-feeding or cross-PE method) in both absolute positional encoding and relative positional encoding. In the case of absolute positional encoding, when evaluated using the BLEU score, the second embodiment achieved the highest scores in both median and mean. In the case of relative positional encoding, when evaluated using the BLEU score, the second embodiment achieved the highest score in both median and mean, and the first embodiment achieved the highest score in mean. Furthermore, in the case of absolute positional encoding, when evaluated using the COMET score, the second embodiment achieved the highest scores in both median and mean. Furthermore, in the case of relative positional encoding, when evaluated using the COMET score, the second embodiment achieved the highest scores in both median and mean.

[0095] Next, the results of the translation from Japanese to English are shown in Table 3 below. The experimental method was the same as for the translation from French to German.

[0096] [Table 3]

[0097] In the translation from Japanese to English, the BLEU scores for the first and some parts of the second embodiment were lower than the conventional techniques. On the other hand, the COMET scores for the first and second embodiments were higher than the conventional techniques in both the absolute positional encoding and the relative positional encoding.

[0098] In other words, although the information processing devices of the first and second embodiments did not obtain higher scores than the conventional technology in all cases, it was demonstrated that they could obtain higher scores than the conventional technology depending on the situation. In other words, the effectiveness of the first and second embodiments was confirmed.

[0099] [Evaluation of translation of sentence lengths not appearing in training data] Next, we evaluate the translation score when translating a long sentence that does not appear in the training data.

[0100] Figures 6(A) and 6(B) are graphs showing the translation scores when a model trained on training data of a specific sentence length is evaluated on evaluation data of various sentence lengths. In these graphs, the bar graphs show the number of training samples for each range of sentence length (number of tokens) in the training data. The vertical axis on the left side of the graph represents the number of training samples, expressed in units of 10 to the power of 6. Figure 6(A) shows that the model was trained using only training data with sentence lengths of approximately 40-59 tokens. Figure 6(B) shows that the model was trained using only training data with sentence lengths of approximately 80-99 tokens and training data with sentence lengths of 100 tokens or more.

[0101] The four line graphs in each of Figures 6(A) and (B) represent the average BLEU scores for the case where absolute positional encoding is used in the information processing device 100 of the first embodiment (Absolute (No Cross-PE)), the case where absolute positional encoding is used in the information processing device 200 of the second embodiment (Absolute (With Cross-PE)), the case where relative positional encoding is used in the information processing device 100 of the first embodiment (Relative (No Cross-PE)), and the case where relative positional encoding is used in the information processing device 200 of the second embodiment (Relative (With Cross-PE)). The horizontal axis represents the range of sentence length (token length) of the evaluation data. The vertical axis (right axis) represents the BLEU score. Note that the BLEU score shown in this graph is the median BLEU value across all three experiments.

[0102] In the case of Figure 6(A) (when model training is performed using training data limited to sentences with lengths of 40-59 tokens), there is a tendency for the BLEU score to decrease overall when translating longer sentences (sentences with lengths of 60 tokens or more). However, when the information processing device 200 of the second embodiment (which uses positional encoding in cross-attention) uses relative positional encoding (Relative (With Cross-PE)), the drop in BLEU score for longer sentences (sentences with lengths of 60 tokens or more) is mitigated. In other words, it was found that when the information processing device 200 of the second embodiment (which uses positional encoding in cross-attention) uses relative positional encoding, relatively better results are obtained compared to other cases when translating long sentences that were not used in the training data.

[0103] Furthermore, in the case of Figure 6(B) (when model training is limited to training data with sentence lengths of 80 tokens or more), there is an overall tendency for the BLEU score to decrease when translating sentences with lengths of 59 tokens or less. However, when the information processing device 200 of the second embodiment (which uses positional encoding in cross-attention) uses relative positional encoding (Relative (With Cross-PE)), the drop in BLEU score is mitigated when translating sentences in the range of 20-39 tokens. Furthermore, when relative positional encoding is used, the drop in BLEU score is mitigated when translating sentences in the range of 40-59 tokens, both in the information processing device 100 of the first embodiment (which does not use positional encoding in cross-attention) and the information processing device 200 of the second embodiment (which uses positional encoding in cross-attention).

[0104] As described above, the graphs in Figures 6(A) and (B) reveal the following. In other words, when training is performed by narrowing down the training data to sentences of a specific sentence length, the translation accuracy of sentences of a sentence length that do not appear in the training data tends to decrease. When the information processing device 200 of the second embodiment (which uses positional encoding in cross-attention) uses relative positional encoding (Relative (With Cross-PE)), the deterioration in translation accuracy in this case can be alleviated. In other words, the accuracy is improved. [Industrial Applicability]

[0105] The present invention can be used, for example, in information processing (one example of which is machine translation of natural language sentences), but the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0106] 11 Input Encoding Section 13 Multi-head attention section 14 Addition and normalization processing section 15 Feedforward processing section 16 Addition and normalization processing section 61 Output Encoding Section 63 Multi-head attention unit with mask 64 Addition and normalization processing section 65 Multi-head Attention Department 66 Addition and normalization processing unit 67 Feedforward processing unit 68 Addition and normalization processing unit 69 Linear transformation processing section 70 Softmax function processing section 100 Information processing device 110 Encoder section 112 Addition section 160 Decoder section 162 Addition section 513 Encoder-side multi-head attention layer 515 Encoder-side feedforward layer 563 Decoder-side multi-head attention layer 565 Cross Attention Layer 567 Decoder-side feedforward layer 612 First Positional Encoding Operation Unit (First PE Operation Unit) 662 Second Positional Encoding Operation Unit (Second PE Operation Unit) 771 Third Positional Encoding Operation Unit (Third PE Operation Unit) 772 4th Positional Encoding Operation Unit (4th PE Operation Unit) 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. an encoder unit that obtains intermediate data based on input data; a first positional encoding unit that applies positional encoding to the input data; a decoder unit that determines output data based on the intermediate data; a second positional encoding unit that applies positional encoding to the output data output by the decoder unit; An information processing device comprising: The encoder unit an encoder-side multi-head attention layer that receives the input data that has been subjected to positional encoding as a query (Q) and a key (K), receives the input data that has not been subjected to positional encoding as a value (V), and calculates a weighted sum of the value (V) based on the similarity between the query (Q) and the key (K); an encoder-side feedforward layer that processes a feedforward network based on data output from the encoder-side multi-head attention layer; Equipped with The decoder unit a decoder-side multi-head attention layer that receives the output data that has been subjected to positional encoding as a query (Q) and a key (K), receives the output data that has not been subjected to positional encoding as a value (V), and calculates a weighted sum of the value (V) based on the similarity between the query (Q) and the key (K); a cross-attention layer that receives the intermediate data output from the encoder-side feedforward layer as values ​​(V) and keys (K), receives the data output from the decoder-side multi-head attention layer as a query (Q), and calculates a weighted sum of the values ​​(V) based on the similarity between the query (Q) and the key (K); an encoder-side feedforward layer that processes a feedforward network based on data output from the cross-attention layer; Equipped with Information processing device.

2. The decoder unit a third positional encoding unit that applies positional encoding to the intermediate data output from the encoder-side feedforward layer; a fourth positional encoding unit that applies positional encoding to data output from the decoder-side multi-head attention layer; Furthermore, The cross-attention layer comprises: The intermediate data to which positional encoding has not been applied is input as a value (V), The intermediate data that has been subjected to positional encoding is input as a key (K), The data output from the decoder-side multi-head attention layer and subjected to positional encoding is input as a query (Q); The information processing device according to claim 1 .

3. The internal parameters are updated by performing error back propagation based on a difference between estimated data, which is output data estimated based on the input data included in the learning data, and the correct answer data, which is a pair of input data and correct answer data, which is a correct answer to output data, based on the learning data. The information processing device according to claim 1 .

4. the input data is data of a token string of a source language sentence, the output data is data of a token sequence of a target language sentence corresponding to the source language sentence; The information processing device according to claim 1 .

5. an encoder unit that obtains intermediate data based on input data; a first positional encoding unit that applies positional encoding to the input data; a decoder unit that determines output data based on the intermediate data; a second positional encoding unit that applies positional encoding to the output data output by the decoder unit; A program that causes a computer to function as an information processing device comprising: The encoder unit an encoder-side multi-head attention layer that receives the input data that has been subjected to positional encoding as a query (Q) and a key (K), receives the input data that has not been subjected to positional encoding as a value (V), and calculates a weighted sum of the value (V) based on the similarity between the query (Q) and the key (K); an encoder-side feedforward layer that processes a feedforward network based on data output from the encoder-side multi-head attention layer; Equipped with The decoder unit a decoder-side multi-head attention layer that receives the output data that has been subjected to positional encoding as a query (Q) and a key (K), receives the output data that has not been subjected to positional encoding as a value (V), and calculates a weighted sum of the value (V) based on the similarity between the query (Q) and the key (K); a cross-attention layer that receives the intermediate data output from the encoder-side feedforward layer as values ​​(V) and keys (K), receives the data output from the decoder-side multi-head attention layer as a query (Q), and calculates a weighted sum of the values ​​(V) based on the similarity between the query (Q) and the key (K); an encoder-side feedforward layer that processes a feedforward network based on data output from the cross-attention layer; Equipped with program.

Images