Information processing device, information processing method, and program

Abstract

This information processing device performs predetermined processing using a neural network model and comprises an inference unit that performs the predetermined processing using the model. The model includes a positional encoding unit that calculates relative positional information of each token in a token string using a wavelet function, and an attention mechanism that calculates a latent representation of the token string using the positional information.

Description

Information processing device, information processing method, and program

[0001] This invention relates to a technique for position coding in neural network models.

[0002] Models such as Transformer are widely used in various generative tasks. These models employ positional encoding, which adds positional information of each token in the input sequence to an embedding vector (for example, Non-Patent Documents 1-5).

[0003] Generally, computational resource constraints during the model's training phase (e.g., pre-training phase) impose limitations on the length of input sequences during training. This constraint causes a decrease in model performance when processing sequences longer than those processed during training during inference. This performance degradation is known to be mainly due to the ineffectiveness of position coding when processing sequences exceeding the maximum length of sequences used during training during inference.

[0004] TRAIN SHORT, TEST LONG: ATTENTION WITH LINEAR BIASES ENABLES INPUT LENGTH EXTRAPOLATION, ICLR 2022A Length-Extrapolatable Transformer, 20 Dec 2022ROFORMER: ENHANCED TRANSFORMER WITH ROTARY POSITION EMBEDDING, 8 Nov 2023Simple Local Attentions Remain Competitive for Long-Context Tasks, Human Language Technologies, pages 1975 - 1986, July 10-15, 2022Attention Is All You Need, 31st Conference on Neural Information Processing Systems (NIPS 2017)

[0005] There are two types of position coding methods: absolute position representation and relative position representation. Absolute position representation indicates the distance (position) of each token from the beginning of a sequence (e.g., a sentence). A challenge with absolute position representation is that it is difficult to represent positions that exceed the maximum length during training.

[0006] On the other hand, relative position representation expresses the distance (relative position) between any two tokens in an input sequence. Relative position representation makes it possible to represent the positions of tokens that exceed the predefined maximum length of the input sequence. However, there is a problem that performance degrades when the length of the sequence significantly exceeds the maximum length. Non-patent document 1 (ALiBi) discloses a technology aimed at solving this problem. However, it is desirable to improve accuracy further.

[0007] This invention has been made in view of the above points, and aims to provide a position coding technique that improves accuracy compared to conventional techniques in neural network models.

[0008] According to the disclosed technology, an information processing device is provided that performs predetermined processing using a neural network model, comprising an inference unit that performs the predetermined processing using the model, the model comprising: a position coding unit that calculates relative position information of each token in a token sequence using a wavelet function, and an attention mechanism that calculates a latent representation of the token sequence using the position information.

[0009] According to the disclosed technology, a position coding technique is provided that offers improved accuracy compared to conventional techniques in neural network models.

[0010] This is a diagram showing an example configuration of the information processing device 100. This is a flowchart explaining the operation of the information processing device 100. This is a diagram showing an example configuration of the information processing device 200. This is a flowchart explaining the operation of the information processing device 100. This is a diagram showing an example configuration of model 300. This is a diagram showing a Transformer model. This is a diagram explaining absolute position representation and relative position representation. This is a diagram showing an example of a Riker wavelet. p ｉｊThis figure shows an image of how the key is added. This figure shows an example of the wavelet function used in the proposed method. This figure shows the experimental results of the extrapolation experiment. This figure shows the experimental results of the interpolation experiment. This figure shows an example of the attention map of the proposed method. This figure shows an example of the waveform of each wavelet function. This figure shows the experimental results for each wavelet function. This figure shows an example of the hardware configuration of the device.

[0011] Hereinafter, embodiments of the present invention (this embodiment) will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below.

[0012] (Outline of the Embodiment) In this embodiment, the information processing device 200, described later, uses a model composed of a neural network to perform the task of generating (inferring) sentences longer than the maximum number of tokens (which may also be called the maximum sequence length) determined during training. Generating sentences longer than the maximum number of tokens determined during training may also be called "extrapolation." Such a task is called TSTL (Train Short, Test Long). Here, "training" is assumed to mean "pre-training," but is not limited to "pre-training." Also, sentences may be referred to as text.

[0013] If the maximum number of tokens determined during training is 512, then tasks that generate sentences longer than the maximum number of tokens determined during training include, for example, "a task that takes a sentence of length 512 as input and generates a sentence that follows it," and "a task that takes a sentence longer than 512, for example, 1024 as input, and generates a sentence that follows it."

[0014] In this embodiment, it is assumed that a Transformer-based model is used as the model described above; however, the base model is not limited to the Transformer. Because the Transformer processes token sequences in parallel, the accuracy decreases as the token sequence lengthens. In particular, during inference, the accuracy drops significantly when a longer token sequence than that used during training is input.

[0015] One of the conventional technologies for solving the above problems is ALiBi, disclosed in Non-Patent Document 1. ALiBi uses a linear bias to represent position. While this method is effective for extrapolation, its simple positional representation can lead to decreased accuracy in tasks that tend to deal with short sentences (e.g., translation). Furthermore, although ALiBi is effective for extrapolation, it is desirable to achieve higher performance.

[0016] In this embodiment, the above problem is solved by employing position coding using wavelet functions, which represents the position using a waveform within a certain range. In this method, the localized waveform is designed based on a wavelet function. Details will be described later.

[0017] (Example of device configuration) The following describes an example of the device configuration in this embodiment. In this embodiment, separate devices are used for the learning phase and the inference phase, but the same device may be used for both the learning phase and the inference phase. Also, the term "learning" below is assumed to refer to "pre-training," but is not limited to "pre-training."

[0018] <Device Configuration in the Learning Phase> Figure 1 shows an example configuration of the information processing device 100 that performs learning. The information processing device 100 that performs learning may also be called a learning device.

[0019] As shown in Figure 1, the information processing device 100 comprises an input unit 110, a learning unit 120, and an output unit 130. The learning unit 120 holds a model 300 to be learned.

[0020] The operation of the information processing device 100 will be explained according to the steps in the flowchart in Figure 2. In S101, training data is input from the input unit 110. The training data is a set of pairs of input data to the model (e.g., text) and correct output data from the model.

[0021] In S102, the learning unit 120 inputs the input data to the model 300 and trains the model 300 by updating its parameters so that the error between the output from the model 300 and the correct data is minimized. The tasks that the model 300 performs during training are not limited to specific tasks, but one example of a task during training is predicting the word following an input text.

[0022] When training is complete, in S103, the output unit 130 outputs the trained model 300 (specifically, model parameters, hyperparameters, etc.). The trained model 300 is used in the inference phase.

[0023] <Device Configuration in the Inference Phase> Figure 3 shows an example configuration of the information processing device 200 that performs inference. The information processing device 200 that performs inference may also be called an inference device.

[0024] As shown in Figure 3, the information processing device 200 comprises an input unit 210, an inference unit 220, and an output unit 230. The inference unit 220 holds a trained model 300. Alternatively, the inference unit 220 can be considered as the model 300 itself. The trained model 300 may be the model itself trained by the aforementioned information processing device 100, or it may be a model that has been fine-tuned.

[0025] The operation of the information processing device 200 will be explained according to the steps in the flowchart of Figure 4. In S201, input data (e.g., text) is input from the input unit 210. In S202, the inference unit 220 inputs the input data to the model 300 and obtains output data from the model 300. The output data is passed to the output unit 230. In S203, the output unit 230 outputs the output data.

[0026] For example, if model 300 is a model trained to translate from a first language to a second language, then model 300 will receive text in the first language as input and output text in the second language.

[0027] (Example of model configuration) Figure 5 shows an example of the configuration of the model 300 that is trained by the learning unit 120 and used by the inference unit 220.

[0028] As shown in Figure 5, Model 300 includes a processing layer A310, a attention mechanism 320, a position coding unit 330, and a processing layer B340. In this embodiment, Model 300 is assumed to be a model having a structure similar to that of a Transformer, but is not limited to this. For example, Model 300 may be a model consisting only of the encoder in a Transformer, or a model consisting only of the decoder in a Transformer.

[0029] The processing layer A310 is a layer that includes an input layer (which may also be called an embedded layer). The attention mechanism 320 has the same configuration as the multi-head attention mechanism in the Transformer. However, the attention mechanism 320 may be a single attention mechanism. The attention mechanism may also be called an attention mechanism.

[0030] The position coding unit 330 performs position coding on the input sequence using a wavelet function based on a relative position representation method. Details will be described later. Processing layer B is a layer that includes the output layer. Note that the function of the position coding unit 330 may be included in the attention mechanism 320.

[0031] (Specific Examples of Models) Specific examples of Model 300 used in the information processing devices 100 and 200 will be described. Here, a general Transformer model will be described as an example of Model 300. A specific example of Model 300 in this embodiment is a model in which the position coding method in a general Transformer model is changed to a method using wavelet functions.

[0032] Figure 6 shows a configuration example of the Transformer model. The configuration shown in Figure 6 is the one disclosed in Non-Patent Document 5 (Attention is all you need). As shown in Figure 6, the Transformer model includes Input Embedding, Output Embedding, Positional Encoding, Multi-Head Attention, Feed Forward, etc. The outlines of these layers / mechanisms are as follows. Since each layer / mechanism is an existing technology, only its outline will be described.

[0033] (1) Input Embedding, Output Embedding Input Embedding and Output Embedding are called embedding layers. The embedding layer performs a process of receiving an input sentence (x ｉ ) Ｉ ｉ＝１ = (x １ ,... x Ｉ ) that has been divided into processing units and obtaining corresponding vectors for each processing unit. In the present embodiment, the processing unit is a token. The token may be referred to as a "word".

[0034] More specifically, the embedding layer uses the vector x ｉ to obtain a d ｉ -dimensional vector e ｍｏｄｅｌ corresponding to the character string w ｉ for each processing unit. e ｉ is a conversion of the token into a real-valued vector of a predetermined dimension (word embedding). The same process is performed for the encoder and the decoder with respect to the processing of the embedding layer.

[0035] (2) Positional Encoding The Positional Encoding (position encoding) of the Transformer is a mechanism that represents the position information of each token in the sentence input to the model by an absolute position representation method. In the configuration shown in Figure 6, a method of adding a value indicating the position to the word embedding (SPE: Sinusoidal Positional Encoding) is used.

[0036] Other position coding techniques include adding bias to the attention score (Non-Patent Literature 1) and multiplying the rotation matrix by the query matrix and key matrix (RoPE: Non-Patent Literature 3). Since the model processes data in parallel, a separate mechanism for representing position is necessary.

[0037] (3) Multi-Head Attention Multi-Head Attention is called a multi-head attention mechanism. A multi-head attention mechanism has multiple attention mechanisms (heads). Each attention mechanism has the function of calculating the relationships between tokens in the input sequence.

[0038] More specifically, the multi-head attention mechanism calculates the latent representation by combining multiple scaled dot-product attention mechanisms. The scaled dot-product attention mechanism calculates the latent representation using the formula shown in "Equation 1" below.

[0039] As shown in the formula above, Attention(Q,K,V) is calculated from the vectors Q (Query), K (Key), and V (Value). T This is called the attention score. The attention score may also be called the relevance score. The above Q, K, and V each have a vector for each token obtained by multiplying the embedded input sequence by a linear transformation matrix.

[0040] More specifically, for example, an embedding sequence x of length N, where each token is embedded in a D-dimensional vector. １ , x ２ ,.... ,x Ｎ When applying an attention mechanism (in this embodiment, a self-attention mechanism is assumed) to x ｉ Query embedding for q ｉ Let x ｉ Key embedding for k ｉ Let x ｉ Value embedding for v ｉ Toshi, W ｑ , W ｋ , Wｖ If we consider each of these to be a D×D matrix, then q ｉ , k ｉ , v ｉ Each of them is q ｉ = x ｉ W ｑ , k ｉ = x ｉ W ｋ , v ｉ = x ｉ W ｖ It is calculated as follows.

[0041] Attention score a of the j-th token as seen from the i-th token ｉｊ This is calculated using the dot product as follows: a ｉｊ =softmax(q ｉ k ｉ Ｔ (√D).

[0042] The multi-head attention mechanism calculates the latent representation using the following formula.

[0043] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W 0 head i =Attention(QW i Q ,KW i K VW i V ) These W i Q ,、W i K , W i V This is a parameter matrix, and these weights are learned during training. In the above MultiHead, Q, K, and V are the embedding vectors (matrices) of the input token sequence. In the multi-head gaze mechanism, latent representations are calculated for each head, and they are concatenated to obtain the weights W. 0 By multiplying by this, the input is returned to the same dimension, and the processing result is output. If the number of dimensions of token embedding is D and the number of heads is M, then the query embedding q in each head in the multi-head attention mechanism. ｉ , key embedded k ｉ, and value embedding v ｉ The number of dimensions for each is D / M. For example, if the number of dimensions for token embedding is 1024 and the number of heads is 8, the number of dimensions for the vector in each head will be 1024 / 8 = 128.

[0044] (4) Feed Forward Layer The feedforward layer is applied independently to each latent representation vector of each token output by the multi-head attention mechanism.

[0045] (Regarding conventional methods of location coding) Next, we will explain examples of conventional techniques in location coding. Since each of these is an existing technology, only an overview will be provided.

[0046] <SPE (Sinusoidal Positional Encoding)> SPE is a positional coding method used in the aforementioned Non-Patent Document 5 (Attention is all you need). In SPE, the absolute position of each token in a sentence is expressed as the position of the token from the beginning of the sentence. Specifically, the position is expressed using an equation based on sine waves and cosine waves, and the information of that position is added to the word embedding.

[0047] <RPR (Relative Position Representation)> RPR is a method for representing the relative position of tokens within a sentence. RPR is a method disclosed in the reference “Self-Attention with Relative Position Representations” (In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468). In RPR, relative position is represented by multiplying positional information with the query, rather than by word embeddings. Note that this method is equivalent to adding positional information to the key.

[0048] Here, we will explain absolute position representation and relative position representation with reference to Figure 7. Figure 7(a) shows absolute position representation, where the absolute position of each token is shown as its position (distance) from the beginning of the sentence. Figure 7(b) shows relative position representation, where the relative position between two tokens is expressed. For example, in Figure 7(b), the vertical axis represents tokens a, b, c, d, and e from top to bottom, and the horizontal axis represents tokens a, b, c, d, and e from left to right. In this case, for example, it expresses things like "the distance between a and a is 0", "b is at a position of +1 relative to a", and "a is at a position of -1 relative to b". In relative position representation, in most cases, the positions of 16 tokens before and after a given token are expressed.

[0049] <RoPE (Rotary Position Embedding)> RoPE, disclosed in Non-Patent Document 3, is a type of relative position representation method that uses a two-dimensional rotation matrix to represent position. Currently, RoPE offers the best accuracy for text generation tasks such as translation. RoPE-based position coding has also been proposed.

[0050] <ALiBi (Attention with Linear Bias)> ALiBi, disclosed in Non-Patent Document 1, is a position coding method effective for the TSTL task. This method does not use position embeddings for position representation, but rather uses a linear bias as shown in equation 2 below. i By adding this to the attention score of each head, relative position representation is achieved. i It is calculated as shown in the following equation (Mathematics 3).

[0051]

[0052] Here, q ｉ ∈R １×ｄ (1 ≤ i ≤ L) is the i-th query of a statement of length L, K ∈ R ｉ×ｄ＿ｈｅａｄ is q ｉ The key for d ｈｅａｄThis is the number of dimensions for each head. For the multi-head attention mechanism, the value of the inclination m for each head is m = 1 / 2 ｎ The calculation is performed, and when the number of heads is 8, m = 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32, 1 / 64, 1 / 128.

[0053] ALiBi has the characteristic that its perplexity decreases as the length of the extrapolated text increases. Perplexity is an evaluation metric for language models that calculate the probability of token occurrence; a lower value indicates better performance.

[0054] (Regarding the location coding method in this embodiment) In long sentences such as documents, not only local information but also information about words distributed throughout the entire sentence is important. The ALiBi described above has the problem of not being able to capture this information sufficiently, resulting in insufficient performance.

[0055] Therefore, this embodiment describes a location representation method that can collect local and global information regardless of the sentence length L.

[0056] In this embodiment, the position coding unit 330 represents relative position using a wavelet function that represents local waves. By changing the localization window of the wavelet for each dimension using the scale parameter a described later, it is possible to collect both local and global information. This localization window is independent of the sentence length L. Furthermore, to avoid affecting the receptive field of self-attention or the gradient of the softmax function, the position representation is added to the key rather than the attention score. This makes it possible to always focus on characteristic tokens.

[0057] <About Wavelet Functions> This section provides an overview of wavelet functions. A wavelet is a wave that decays rapidly locally and approaches zero. A function ψ defined on the real numbers R is defined in the space L of square-integrable functions. ２ A wavelet function is called a wavelet function when it belongs to (R) and satisfies the following conditions.

[0058] Frequency analysis is performed on the time axis t, where the wavelets are shifted or scaled as shown below ψ ａ，ｂ This can be done by using (t) as a basis function and taking the inner product with the signal f(t). This is called the wavelet transform.

[0059] In the equation "Equation 5" above, b is the shift parameter and a (>0) is the scale parameter. The Fourier transform decomposes a signal into sine and cosine waves of various frequencies, while the wavelet transform decomposes a signal into shifted and scaled wavelets using the above wavelets as basis functions.

[0060] The Fourier transform periodically transforms waves, making it effective for standing waves. In contrast, the wavelet transform decomposes waves into local waves, making it easier to capture the characteristics of irregular and non-stationary waves. Typical wavelets include Riker wavelets, Moret wavelets, and Gaussian-based wavelets.

[0061] Figure 8 shows an example of a Riker wavelet. As shown in Figure 8, the local wave width increases as the scale parameter a increases.

[0062] <Proposed Method> The position coding method using wavelet functions according to this embodiment is called the "proposed method." In this embodiment, the position coding unit 330 calculates the relative position value between tokens using wavelet functions. This value is used to calculate the attention score for each head in the attention mechanism 320.

[0063] More specifically, the attention mechanism 320 uses the following formula "Equation 6" to determine the attention score α of the j-th token as seen from the i-th token in each head. ｉｊ Calculate α. ｉｊ This may also be called the attention score from the i-th token to the j-th token, the attention score between the i-th token and the j-th token, the attention score related to the i-th token and the j-th token, and so on.

[0064] In the above equation, q ｉ This is the i-th query (q) of a sentence (token sequence) of length L. ｉ ∈R １×ｄ ) (1 ≤ i ≤ L), and k ｊ is, q ｉ Key sequence K∈R ｉ×ｄ＿ｈｅａｄ This is the j-th key in [the system]. ｈｅａｄ This represents the number of dimensions for each head.

[0065] p ｉｊ ∈R ｄ＿ｈｅａｄ p is the relative position of the i-th token to the j-th token. ｉｊ This can also be called the distance from the i-th token to the j-th token, the assumed position between the i-th token and the j-th token, the relative position between the i-th token and the j-th token, the distance between the i-th token and the j-th token, and so on.

[0066] The position coding unit 330 of this embodiment uses a wavelet function as shown in "Equation 7" and "Equation 8" to perform p ｉｊ To calculate this, we substitute "i-j" into the wavelet function of "Equation 8". ｉｊ The value is calculated.

[0067]

[0068] In the above equation, a is the scale parameter of the wavelet function, and b is the shift parameter. Note that the amplitude is different from that shown in "Equation 5" above. In a normal wavelet, the amplitude also differs depending on the scale parameter a, but in the proposed method, the amplitude is always the same.

[0069] As in "Math 6", p was calculated using the wavelet function. ｉｊThe image added to the key is shown in FIG. 9. FIG. 9 shows the case where four wavelet functions are composed of four types of a and one type of b. In FIG. 9, for example, as the value of the position of the first dimension of each vector representing a token, a wavelet function with a = 0 and b = 0 is used, but the value is different for each token. The same applies to the positions of the dimensions other than the first dimension of the vector. Regarding the graph of each wavelet function shown on the lower side of FIG. 9, the horizontal axis corresponds to t in "Equation 8", and the value of the wavelet function (p ｉｊ ) is determined according to the value of t.

[0070] In the proposed method, for the scale parameter a, n ｈｅａｄ patterns are prepared, and for the shift parameter b, d ｈｅａｄ / n ｈｅａｄ patterns are prepared, and by combining these, d ｈｅａｄ wavelet functions are created to calculate p ｉｊ . n ｈｅａｄ is the number of heads.

[0071] The scale parameter a is selected so that the local window of the wave does not exceed the maximum length L ｍａｘ . For example, when the number of head dimensions is d ｈｅａｄ = 128, eight scale parameters a and 16 shift parameters b are used, and 8 × 16 = 128 types of wavelets are used. The actual parameters are as follows.

[0072] d ｈｅａｄ / n ｈｅａｄ shift parameters and n ｈｅａｄ scale parameters can be combined to create d ｈｅａｄ wavelets. Therefore, it is possible to represent the position based on different wavelets in all dimensions.

[0073] For example, when the number of dimensions of each head is 128 dimensions, 128 wavelet functions obtained by combining eight scale parameters and 16 shift parameters are used. The eight scale parameters are 2 ０ 、2１ ,...,2 ７ The 16 shift parameters are 0, 1, 2, 3, ..., 15. In this specification and in the drawings, the value of the scale parameter 2 ａ It is sometimes written as "a".

[0074] The wavelet function used in this embodiment is the Ricker wavelet. The formula for the Ricker wavelet is formulated as follows:

[0075] Figure 10 shows an example of the wavelet function used in the proposed method. In the example shown in Figure 10, a = 2 ０ ,2 １ ,2 ２ ,2 ３ And 16 different wavelet functions with b = 0, 1, 2, and 3 are shown.

[0076] The scale parameter allows for explicit specification of distance in the positional representation, and the shift parameter enables different wavelet-based positional representations for each dimension. The distance in the positional representation does not depend on the length of the sentence.

[0077] In this embodiment, the computational complexity is reduced by transforming the formula in "Equation 6" as follows.

[0078] By performing this transformation, the computational complexity can be expressed as O(batch × n × length) ２ ×d ｈｅａｄ +length ２ ×d ｈｅａｄ ) can be reduced to ). Here, batch is the batch size, n is the number of heads, length is the number of tokens, and d ｈｅａｄ This represents the number of dimensions for each head.

[0079] <Flow of position representation> When the number of dimensions of each head is 128, the flow of position representation performed by the position coding unit 330 is as follows.

[0080] First, the position coding unit 330 sets a=2 ０ , 2 １ ,...,2 ７A total of 128 wavelet functions are prepared by combining b = 0, 1, 2, 3, ..., 15. Specifically, for example, the position coding unit 330 reads a wavelet function from a storage device such as the memory of the information processing device 100 or 200, and prepares 128 wavelet functions by setting a and b to the wavelet function.

[0081] Next, the position coding unit 330 calculates the value of the wavelet function p for the relative position t = i - j for each of the 128 wavelet functions for each head. ｉｊ The position coding unit 330 calculates the value and passes the calculated value to the attention mechanism 320.

[0082] The attention mechanism 320 is, according to formula "Equation 10", (p ｉｊ ) Ｔ query q ｉ Apply to q ｉ (p ｉｊ ) Ｔ to q ｉ k ｊ Ｔ Adding this to the attention score α ｉｊ The attention mechanism 320 calculates the attention score α for each i and each j. ｉｊ Using this, the latent representation of the input token sequence input to model 300 is calculated and output.

[0083] The functions of the position coding unit 330 may also be included in the attention mechanism 320. In that case, the attention mechanism 320 performs each of the above processes.

[0084] (Regarding the experiment) An experiment was conducted to confirm the effectiveness of the technology in this embodiment, and the details of the experiment are described below. The names of the references (excluding "non-patent literature") mentioned below are listed together at the end of the specification.

[0085] <Experimental Setup> To verify the extrapolation performance of the proposed method, an extrapolation experiment was conducted. The dataset used was the WikiText-103 dataset. The WikiText-103 dataset consists of English Wikipedia articles with over 130 million tokens. A Transformer-based language model was used for comparative evaluation. The number of dimensions of word embeddings was d. ｍｏｄｅｌ The number of heads is 1024, the number of heads n is 8, the number of head dimensions for each head is 128, and the number of layers is 16.

[0086] The parameter settings used in the extrapolation experiment were the same as those disclosed in Non-Patent Document 1 for ALiBi. The number of training epochs was set to 205, the batch size to 9216, and the learning rate to 1.0. The learning process was updated every 16000 steps at a rate of 1e-7. The implementation used the code based on fairseq (Ott et al., 2019) provided in Non-Patent Document 1, and all hyperparameters were set to the same values ​​as in Non-Patent Document 1. Perplexity was used as the evaluation metric, and the maximum length of the input sequence during training was L. ｍａｘ = 512.

[0087] <Comparison Method> In addition to ALiBi, the following position representation methods were also included in the comparison.

[0088] ・Sinusoidal (Vaswani et al., 2017) (Non-Patent Literature 5) - Absolute position representation using sine waves ・T5 Bias (Raffel et al., 2020) - Relative position representation using learnable embeddings extended to 128 tokens ・RoPE (Su et al., 2021) (Non-Patent Literature 3) - Absolute position representation based on rotation matrices (relative position can also be captured due to the properties of the rotation matrices) ・Trans-XL (Dai et al., 2019) - Relative position representation using sine waves ・NoPE (Kazemnejad et al., 2023) - A method that does not provide any position information ・XPOS (Sun et al., 2023) (Non-Patent Literature 2) - Position representation using exponentially decaying RoPE <Experimental results of extrapolation task> The experimental results are shown in Figure 11. As shown in Figure 11, when generating (inferring) sentences with a number of tokens longer than 512 (the maximum length during training), the proposed method (WaveletPos) has the lowest perplexity at all lengths. In other words, the proposed method performs best.

[0089] <Interpolation Results> Figure 12 shows the experimental results of the interpolation task, which generates sentences shorter than the maximum learning length. As shown in Figure 12, the proposed method yields the best results even when generating short sentences.

[0090] <Visualization Analysis> Figure 13 shows the attention map (heatmap of attention scores) after applying the softmax function to the proposed method. The maximum sequence length during training is L. ｍａｘ = 512, and the sequence length during inference is L = 1012. From left to right, the attention maps for the 1st, 2nd, and 4th heads are shown. Scores greater than 0.01 are indicated by black dots. Words that were consistently noticed by all heads are shown with lines extending from the attention map.

[0091] First, a key feature of the proposed method is its ability to assign attention to specific tokens. The words to which attention is assigned are important "" tokens in a sentence, the first token, and the subject of a sequence, such as "he".

[0092] In ALiBi, the receptive field of tokens far from the center is limited, whereas the proposed method pays attention to even distant tokens. Furthermore, as mentioned above, it pays attention to characteristic words. In other words, it can be considered that the proposed method can represent locations over a wide range without losing information about important words.

[0093] (Regarding other wavelet functions) As mentioned above, the Ricker wavelet function is used as the wavelet function in this embodiment. However, the wavelet function that can be used in the technology according to this embodiment is not limited to the Ricker wavelet function, and other wavelet functions can also be used.

[0094] Therefore, we conducted experiments to verify whether other wavelet functions could produce the same effect as the Ricker wavelet function. The wavelet functions we examined were the Gaussian-based wavelet function (equation "Equation 12" below) and the Moret wavelet function (equation "Equation 13" below).

[0095]

[0096] Gaussian-based wavelet functions are based on Gaussian functions. The Moret wavelet function is a product of the Gaussian function and the cosine wavelet function. Figure 14 shows examples of waveforms for each wavelet function. Figure 14 shows the case where the scale parameter a = 16 and the shift parameter b = 0. Note that ψ Ｍｏｒｌｅｔ When using (t / a), the frequency of the cosine wavelet is not affected by the scale parameter a.

[0097] Using each of the wavelet functions described above, extrapolation experiments were performed under the same experimental setup as described earlier. The experimental results are shown in Figure 15. As shown in Figure 15, the Ricker wavelet showed the lowest perplexity, followed by the Gaussian wavelet.

[0098] (Example Hardware Configuration) Any of the devices (information processing devices 100, 200) described in this embodiment can be realized, for example, by having a computer execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0099] In other words, the device can be realized by using hardware resources such as the CPU and memory built into a computer to execute a program corresponding to the processing performed by the device. The program can be recorded on a computer-readable recording medium (such as portable memory), saved, and distributed. It can also be provided via a network, such as the Internet or email.

[0100] Figure 16 shows an example of the hardware configuration of the computer described above. The computer in Figure 16 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., all of which are interconnected by bus B. The computer may also be equipped with a GPU.

[0101] The program that enables processing on the computer is provided on a recording medium 1001, such as a CD-ROM or memory card. When the recording medium 1001 containing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001; it may also be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files and data.

[0102] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when a program startup command is received. The CPU 1004 implements the functions related to the memory device 1003 according to the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) etc., based on a program. The input device 1007 consists of a keyboard and mouse, buttons, or a touch panel, etc., and is used to input various operation commands. The output device 1008 outputs the calculation results.

[0103] (Summary of Embodiments, Effects, etc.) As described above, the technology described in this embodiment makes it possible to realize a position coding technology with improved accuracy compared to conventional technologies in neural network models.

[0104] The following additional information is disclosed regarding the embodiments described above.

[0105] <Notes> (Note 1) An information processing device that performs predetermined processing using a neural network model, comprising: a memory; and at least one processor connected to the memory, wherein the processor performs the predetermined processing using the model, and the model comprises: a position coding unit that calculates relative position information of each token in a token sequence using a wavelet function; and an attention mechanism that calculates a latent representation of the token sequence using the position information. (Note 2) The information processing device according to Note 1, wherein the attention mechanism calculates the latent representation by querying the position information. (Note 3) The information processing device according to Note 1, wherein the attention mechanism has a plurality of heads, and the position coding unit calculates the position information using a number of wavelet functions equal to the number of dimensions of the vector in each head. (Note 4) The information processing device according to Note 3, wherein each wavelet function has a scale parameter and a shift parameter, and the number of dimensions is the number obtained by multiplying the number of scale parameters by the number of shift parameters. (Appendix 5) The position coding unit calculates the position information which is the output of the wavelet function by inputting the distance between two tokens to the wavelet function, as described in Appendix 1. (Appendix 6) An information processing device for training a neural network model, comprising: a memory; and at least one processor connected to the memory, wherein the processor inputs data to the model and adjusts the parameters of the model so that the output from the model for the data becomes the correct data, and the model comprises: a position coding unit that calculates the relative position information of each token in a token sequence using a wavelet function; and an attention mechanism that calculates a latent representation of the token sequence using the position information.(Appendix 7) An information processing method performed by an information processing device that performs predetermined processing using a neural network model, comprising an inference step of performing the predetermined processing using the model, wherein the model calculates relative position information of each token in a token sequence using a wavelet function, and calculates a latent representation of the token sequence using the position information. (Appendix 8) A non-temporary storage medium storing a program for causing a computer to function as an information processing device according to any one of Appendix 1 to 6.

[0106] Although this embodiment has been described above, the present invention is not limited to this specific embodiment, and various modifications and changes are possible within the scope of the gist of the invention as described in the claims.

[0107] [References] Ott et al., 2019: Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. Raffel et al., 2020: Colin Raffel, Noam Shazeer, Adam Roberts, Kather- 694 ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67. Dai et al., 2019: Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57 thAnnual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Association for Computational Linguistics. Kazemnejad et al., 2023: Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. 2023. The impact of positional encoding on length generalization in transformers. In Advances in Neural Information Processing Systems, volume 36, pages 24892-24928. Curran Associates, Inc.

[0108] 100, 200 Information processing device 110, 210 Input unit 120 Learning unit 130, 230 Output unit 220 Inference unit 300 Model 310 Processing layer A 320 Attention mechanism 330 Position coding unit 340 Processing layer B 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. An information processing device that performs predetermined processing using a neural network model, comprising an inference unit that performs the predetermined processing using the model, the model comprising: a position coding unit that calculates relative position information of each token in a token sequence using a wavelet function; and an attention mechanism that calculates a latent representation of the token sequence using the position information.

2. The information processing apparatus according to claim 1, wherein the attention mechanism calculates the latent representation by querying the position information.

3. The information processing apparatus according to claim 1, wherein the attention mechanism has a plurality of heads, and the position coding unit calculates the position information using wavelet functions equal to the number of dimensions of the vector in each head.

4. The information processing device according to claim 3, wherein each wavelet function has a scale parameter and a shift parameter, and the number of dimensions is the number obtained by multiplying the number of scale parameters by the number of shift parameters.

5. The information processing apparatus according to claim 1, wherein the position coding unit calculates the position information, which is the output of the wavelet function, by inputting the distance between two tokens to the wavelet function.

6. An information processing device for training a neural network model, comprising: a learning unit that inputs data to the model and adjusts the parameters of the model so that the output from the model for the data becomes the correct answer data, wherein the model comprises: a position coding unit that calculates relative position information of each token in a token sequence using a wavelet function; and an attention mechanism that calculates a latent representation of the token sequence using the position information.

7. An information processing method performed by an information processing device that performs predetermined processing using a neural network model, comprising an inference step that performs the predetermined processing using the model, wherein the model calculates relative positional information of each token in a token sequence using a wavelet function, and calculates a latent representation of the token sequence using the positional information.

8. A program for causing a computer to function as an information processing device according to any one of claims 1 to 6.

Images