Learning device, estimation device, learning method, estimation method, and program

Abstract

To suppress deterioration in accuracy of series processing.SOLUTION: A control unit that performs learning of a learning target model that is a mathematical model for estimating, based on first time-series data in which an amount at a time t that is a t-th time is a D-dimensional vector xt, second time-series data that is time-series data identical to or different from the first time-series data, the learning target model estimates data at time t in the second time-series data based on a hidden vector ht obtained based on the vector xt and the hidden vector ht-1, and the hidden vector ht is a D-dimensional vector and is a vector in which one of a value of a d-th element in a D-dimensional feature vector of the vector xt and a value of a d-th element in the hidden vector ht-1 determined according to a predetermined rule is a value of the d-th element.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a learning device, an estimation device, a learning method, an estimation method, and a program. [Background technology]

[0002] There has been growing interest in mathematical models for sequential processing such as translation, and for example, sequential processing using recurrent neural networks (RNNs) has been proposed. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Albert Gu and Tri Dao. "Mamba: Linear-time sequence modeling with selective state spaces." arXiv preprint arXiv:2312.00752 (2023). Summary of the Invention [Problem to be solved by the invention]

[0004] However, in RNNs, the hidden state is constantly transformed at each time point. Therefore, if the input is long, the RNN may undergo too many transformations, leading to accumulated errors and incorrect updates. As a result, the accuracy of sequence processing may decrease.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique for suppressing deterioration in accuracy of series processing. [Means for solving the problem]

[0006] In one aspect of the present invention, the quantity at time t, which is the tth time (t is an integer equal to or greater than 1), is expressed as a D-dimensional vector x t(D is an integer equal to or greater than 1), and a control unit that performs training of a training model that is a mathematical model that estimates second time series data that is the same as or different from the first time series data, based on the first time series data, t and the hidden vector h t-1 The hidden vector h obtained based on t The data at time t in the second time series data is estimated based on the hidden vector h t is a D-dimensional vector, and the vector x t The value of the dth element (d is an integer between 1 and D) in the D-dimensional feature vector of t-1 and the value of the d-th element in x1 is determined according to a predetermined rule, and the hidden vector h1 at the first time instant is a D-dimensional feature vector of the vector x1.

[0007] In one aspect of the present invention, the quantity at time t, which is the tth time (t is an integer equal to or greater than 1), is expressed as a D-dimensional vector x t (D is an integer equal to or greater than 1), and a control unit that performs training of a training model that is a mathematical model that estimates second time series data that is the same as or different from the first time series data, based on the first time series data, t and the hidden vector h t-1 The hidden vector h obtained based on t The data at time t in the second time series data is estimated based on the hidden vector h t is a D-dimensional vector, and the vector x t The value of the dth element (d is an integer between 1 and D) in the D-dimensional feature vector of t-1 and a hidden vector h1 at the first time instant is a D-dimensional feature vector of vector x1, the hidden vector h1 at the first time instant being a D-dimensional feature vector of vector x1.

[0008] In one aspect of the present invention, the quantity at time t, which is the tth time (t is an integer equal to or greater than 1), is expressed as a D-dimensional vector x t (D is an integer equal to or greater than 1), and a control unit that performs training of a training model that is a mathematical model that estimates second time series data that is the same as or different from the first time series data, based on the first time series data, t and the hidden vector h t-1 The hidden vector h obtained based on t The data at time t in the second time series data is estimated based on the hidden vector h t is a D-dimensional vector, and the vector x t The value of the dth element (d is an integer between 1 and D) in the D-dimensional feature vector of t-1 and a hidden vector h1 at the first time instant is a D-dimensional feature vector of vector x1, the hidden vector h1 being ...

[0009] In one aspect of the present invention, the quantity at time t, which is the tth time (t is an integer equal to or greater than 1), is expressed as a D-dimensional vector x t (D is an integer equal to or greater than 1), and a control unit that performs training of a training model that is a mathematical model that estimates second time series data that is the same as or different from the first time series data, based on the first time series data, t and the hidden vector h t-1 The hidden vector h obtained based on t The data at time t in the second time series data is estimated based on the hidden vector h t is a D-dimensional vector, and the vector x t The value of the dth element (d is an integer between 1 and D) in the D-dimensional feature vector of t-1and a hidden vector h1 at the first time instant is a D-dimensional feature vector of vector x1, the hidden vector h1 at the first time instant being a D-dimensional feature vector of vector x1, and the hidden vector h1 at the first time instant is a D-dimensional feature vector of vector x1.

[0010] One aspect of the present invention is a program for causing a computer to function as the learning device described above.

[0011] One aspect of the present invention is a program for causing a computer to function as the above-described estimation device. [Effects of the Invention]

[0012] The present invention makes it possible to suppress deterioration in accuracy of series processing. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is an explanatory diagram illustrating an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning device according to an embodiment. [Figure 3] 10 is a flowchart showing an example of a flow of processing executed by a learning device according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of an estimation apparatus according to an embodiment. [Figure 5] 1 is a flowchart showing an example of a flow of processing executed by an estimation device according to an embodiment. [Figure 6] FIG. 10 is a diagram showing an example of an experimental result in the embodiment. [Figure 7] FIG. 10 is a diagram showing an example of a hidden vector obtained according to category-dependent rules in a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0014] (Embodiment) 1 is an explanatory diagram illustrating an information processing system 100 according to an embodiment. The information processing system 100 includes a learning device 1 and an estimation device 2.

[0015] The learning device 1 includes a control unit 11 having a processor 91, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit), and a memory 92, which are connected via a bus, and executes a program.

[0016] The control unit 11 executes, for example, a learning process. The learning process involves learning a learning target model, which is a mathematical model of a learning target. The learning may be supervised learning, unsupervised learning, self-supervised learning, or semi-supervised learning. Furthermore, backpropagation may be used in the learning.

[0017] The model to be trained is a D-dimensional vector x t (D is an integer equal to or greater than 1) based on time series data (hereinafter referred to as "first time series data"), which may be the same as or different from the first time series data (hereinafter referred to as "second time series data"). The time series data may be, for example, an English sentence such as "I am a..." In this case, for example, a vector x t represents a word. For example, in the sentence "I am a", the vector x1 represents "I", the vector x2 represents "am", and the vector x3 represents "a".

[0018] More specifically, the model to be trained is a vector x t Based on this, the data z of the second time series data at time t t It is a mathematical model that estimates the data z t To explain the relationship between the second time series data and the data z, t The ordered set of these is the second time series data.

[0019] The model to be trained is, more specifically, a vector x t and the hidden vector h t-1 The hidden vector h obtained based on t Based on this, the data z at time t in the second time series data t Estimate.

[0020] Hidden vector h t is the value Q (d) t and the value h (d) t-1 It is a D-dimensional vector whose dth element (d is an integer between 1 and D) is determined according to a predetermined rule (hereinafter referred to as "element rule"). (d) t is the vector x t D-dimensional feature vector Q t The value of the dth element in h (d) t-1 is the hidden vector h t-1 is the value of the dth element in the hidden vector h t The value of the dth element of is the value h (d) t is.

[0021] Thus, the value h (d) t is the value Q (d) t and the value h (d) t-1 and is the identity mapping for either

[0022] The hidden vector h1 at the first time is a D-dimensional feature vector of the vector x1.

[0023] The learning model may be a mathematical model represented by a neural network, for example.

[0024] The estimation device 2 performs estimation processing. The estimation processing is processing for making estimations using the learning results obtained by the learning device 1. Using the learning results obtained by the learning device 1 means using the results of learning executed by the control unit 11. The learning results are, specifically, a trained learning object model. Specifically, in the estimation processing, second time series data is estimated based on the first time series data using the trained learning object model.

[0025] The trained mathematical model refers to a mathematical model at the point in time when a predetermined condition for terminating the learning (hereinafter referred to as the "learning termination condition") is satisfied. The learning termination condition may be, for example, that the mathematical model has been updated a predetermined number of times, or that the change in the mathematical model due to the update is smaller than a predetermined change.

[0026] <First example of element rule> An element rule is, for example, the value Q (d) t and the value h (d) t-1 The rule is that the value of the dth element is the one that is not smaller than the other. (d) t The value h (d) t-1 If it is greater than the value Q (d) t is the hidden vector h t The value of the dth element of h (d) t For example, the value Q (d) t The value h (d) t-1 If it is less than the value h (d) t-1 is the hidden vector h t The value of the dth element of h (d) t For example, the value Q (d) t and the value h (d) t-1 If and are the same, the value Q (d) t and the value h (d) t-1One of them is the hidden vector h t The value of the dth element of h (d) t is.

[0027] <Second example of element rule> An element rule is, for example, the value Q (d) t and the value h (d) t-1 The rule is that the value of the dth element is the one that is not greater than the other. (d) t The value h (d) t-1 If it is less than the value Q (d) t is the hidden vector h t The value of the dth element of h (d) t For example, the value Q (d) t The value h (d) t-1 If it is greater than the value h (d) t-1 is the hidden vector h t The value of the dth element of h (d) t Also, for example, the value Q (d) t and the value h (d) t-1 If and are the same, the value Q (d) t and the value h (d) t-1 One of them is the hidden vector h t The value of the dth element of h (d) t is.

[0028] <Explanation using mathematical formulas> Using the formula, the hidden vector h t+1 An example of the hidden vector h t+1 When expressed mathematically, it can be expressed, for example, by the following formula (1).

[0029]

number

[0030] where r(x t+1 ) can be linearly or nonlinearly t+1 Therefore, r(x t+1 ) is the image of the vector x t+1 A D-dimensional feature vector Q t+1 Update(r(x t+1 ),h t ) is expressed as h t+1 the i-th element of h (i) t+1 For r(x t+1 ) and h t This is a process of selecting either the i-th element or the i-th element.

[0031]

number

[0032] where r (i) (x t+1 ) is r(x t+1 ), and u i means a given rule. i =1 means that this given rule is satisfied, and u i = 0 means that this predetermined rule is not satisfied. Therefore, formula (2) is an example of an element rule. i For example, r (i) (x t+1 ) is better (i) t The rule may be that u is not less than u. i For example, r (i) (x t+1 ) is better (i) t The rule may be that the number is not greater than 1.

[0033] <Effects of learning processing> To explain the effect of the learning process, we will explain recurrent neural networks. In recurrent neural networks, the hidden state continues to be recursively transformed from time t to time t+T. Therefore, in recurrent neural networks, there is a risk that memory will deteriorate due to incorrect transformations or the accumulation of errors.

[0034] On the other hand, in the case of learning processing, the risk of memory deterioration is smaller than in recurrent neural networks. This is because, as explained so far, the hidden vector h t+1 The value of the element of h (d) t+1 But the value Q (d) t+1 and the value h (d) t This is because the identity mapping is either one of the two and is not recursively defined. Therefore, the learning process can suppress the deterioration of the accuracy of the sequential processing.

[0035] Incidentally, we have mentioned that backpropagation can be used in learning. Backpropagation is also used in recurrent neural networks, but it is known that there is a risk of gradient disappearance or explosion in recurrent neural networks. The following equation (3) expresses the relationship that the gradient satisfies in backpropagation. As this equation expresses, in backpropagation, the loss L obtained at time t+T is t+T ,t's hidden vector h t The gradient for is obtained. Note that time t+T is the time T hours have passed since time t.

[0036]

number

[0037] Furthermore, equation (3) shows that when backpropagation is used, many differential multiplications are performed according to the chain rule. As mentioned above, in recurrent neural networks, hidden states are transformed recursively. Therefore, in recurrent neural networks, when the sequence length is large, the gradient is backpropagated through a relatively large number of time layers, and the many differential multiplications according to the chain rule described above can result in gradient vanishing or explosion.

[0038] On the other hand, in the case of the learning process, even if backpropagation is used, the risk of gradient vanishing or exploding is smaller than in recurrent neural networks. This is because, as explained so far, the hidden vector h t+1 The value of the element of h (d) t+1 But the value Q (d) t+1 and the value h (d) t This is because it is the identity mapping of either

[0039] We will explain using mathematical formulas why the risk of gradient vanishing or exploding is smaller than in recurrent neural networks. As mentioned above, in the learning process, the hidden vector h t The elements of are obtained by identity mapping. Therefore, the i-th element of the hidden vector at time t can be held in the hidden vector at time t+T. Therefore, in this case, the following equation (4) holds.

[0040]

number

[0041] Since equation (4) holds, the slope of the loss at time t+T shown in equation (3) is h t+T From h t The training process is calculated without multiplication of differentials up to . Therefore, even if backpropagation is used, the risk of gradient vanishing or exploding is smaller than in recurrent neural networks. Therefore, the training process can suppress the deterioration of accuracy in sequential processing.

[0042] Note that the vector x t may be a vector obtained by the Sliding Window Attention mechanism based on the original data. Here, the original data is a time series data of a D-dimensional vector, and the vector at each time t is a vector x t The time series data may be the same as or different from the above.

[0043] The model to be trained is the hidden vector h t+1 Here, the output is the data z at time t+1 of the second time series data. t+1 is.

[0044] Note that the vector x t may be a vector obtained by a recurrent neural network based on the source data.

[0045] The learning model is generated by a recurrent neural network using the hidden vector h t+1 Here, the output is the data z at time t+1 of the second time series data. t+1 is.

[0046] <Example of hardware configuration of learning device 1> 2 is a diagram showing an example of the hardware configuration of the learning device 1 according to the embodiment. The learning device 1 includes a control unit 11 and executes a program, and functions as a device including the control unit 11, an interface unit 12 including a communication interface 121, and a storage unit 13 by executing the program.

[0047] More specifically, the processor 91 reads out a program stored in the storage unit 13 and stores the read out program in the memory 92. When the processor 91 executes the program stored in the memory 92, the learning device 1 functions as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0048] The control unit 11 controls the operation of each functional unit included in the learning device 1. The control unit 11 executes, for example, the learning process as described above. The control unit 11 acquires, for example, information stored in the memory unit 13. Specifically, the process of acquiring the information stored in the memory unit 13 is reading.

[0049] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless communication.

[0050] The external device is, for example, a device that transmits data used in the learning process. In such a case, the interface unit 12 acquires the data used in the learning process by communicating with the device that transmits the data used in the learning process.

[0051] The external device may be, for example, the estimation device 2. In this case, the estimation device 2 can perform estimation using the learning results obtained by the learning device 1 through communication via the interface unit 12.

[0052] Interface unit 12 may be configured to include input devices such as a mouse, keyboard, or touch panel. Interface unit 12 may be configured as an interface that connects these input devices to learning device 1. In this way, the input devices of interface unit 12 accept input of various information to learning device 1 via wired or wireless connections. Note that information does not necessarily have to be input to the communication interface of interface unit 12, but may also be input to the input devices of interface unit 12.

[0053] Interface unit 12 outputs, for example, various types of information. Interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display, and a speaker. Interface unit 12 may be configured as an interface that connects these display devices or speakers to learning device 1. Therefore, interface unit 12 may output, for example, information input to an input device of interface unit 12 as an image or sound.

[0054] The storage unit 13 is configured using a computer-readable storage medium (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores various information generated by the operation of the control unit 11, for example. The storage unit 13 may exist on a cloud, for example.

[0055] 3 is a flowchart showing an example of the flow of processing executed by the learning device 1 in the embodiment. The control unit 11 acquires data to be used in the learning processing (step S101). Next, the control unit 11 executes the learning processing (step S102).

[0056] <Example of hardware configuration of estimation device 2> 4 is a diagram illustrating an example of a hardware configuration of the estimation device 2 according to an embodiment. The estimation device 2 includes a control unit 21 including a processor 93 such as a CPU, GPU, or NPU, and a memory 94, which are connected via a bus, and executes a program. By executing the program, the estimation device 2 functions as a device including the control unit 21, an interface unit 22, and a storage unit 23.

[0057] More specifically, the processor 93 reads out a program stored in the storage unit 23 and stores the read program in the memory 94. The processor 93 executes the program stored in the memory 94, causing the estimation device 2 to function as a device including the control unit 21, the interface unit 22, and the storage unit 23.

[0058] The control unit 21 controls the operation of each functional unit included in the estimation device 2. The control unit 21 executes, for example, an estimation process. The control unit 21 acquires, for example, information stored in the memory unit 23. Specifically, the process of acquiring the information stored in the memory unit 23 is reading.

[0059] The interface unit 22 includes a communication interface for connecting the estimation device 2 to an external device. The interface unit 22 communicates with the external device via wire or wirelessly.

[0060] The external device is, for example, a device that transmits estimation source information in the estimation process. The estimation source information is information based on which estimation is performed in the estimation process. Specifically, the estimation source information is first time series data to be input to a trained learning object model. Having acquired the estimation source information, the control unit 21 executes the estimation process to estimate second time series data based on the first time series data using the trained learning object model. The interface unit 22 acquires the estimation source information in the estimation process by communicating with the device that transmits the estimation source information in such estimation process.

[0061] The external device may be, for example, the learning device 1. In this case, the estimation device 2 can use the learning results obtained by the learning device 1 through communication via the interface unit 22.

[0062] The interface unit 22 may be configured to include input devices such as a mouse, a keyboard, a touch panel, etc. The interface unit 22 may be configured as an interface that connects these input devices to the estimation device 2. In this way, the input devices of the interface unit 22 accept input of various information to the estimation device 2 via wired or wireless connections. Note that information does not necessarily have to be input to the communication interface of the interface unit 22, and may also be input to the input devices of the interface unit 22.

[0063] The interface unit 22 outputs, for example, various types of information. The interface unit 22 includes, for example, a display device such as a CRT display, a liquid crystal display, or an organic EL display, and a speaker. The interface unit 22 may be configured as an interface that connects these display devices or speakers to the estimation device 2. Therefore, the interface unit 22 may output, for example, information input to an input device of the interface unit 22 as an image or sound.

[0064] The storage unit 23 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the estimation device 2. The storage unit 23 stores various information generated by the operation of the control unit 21, for example. The storage unit 23 may exist on a cloud, for example.

[0065] 5 is a flowchart showing an example of the flow of processing executed by the estimation device 2 in the embodiment. The control unit 21 acquires information on the estimation source (i.e., first time-series data) (step S201). The control unit 21 executes estimation processing (step S202).

[0066] <Experimental Results> The effectiveness of the learning process has been experimentally confirmed and will be explained below. The specific learning model used in the experiment (hereinafter referred to as the "experimental model") will be explained. In explaining the experimental results, for the sake of simplicity, the hidden vector h t is expressed by the following equation (5). In the experiment, the hidden vector h t Therefore, in the explanation of the experiment, they are collectively called the tensor H t Therefore, equation (5) represents the N global tokens as a combined tensor.

[0067]

number

[0068] The experimental model uses MemMHA(·) to calculate the output y t+1 Note that MemMHA(·) is x t+1 is linearly transformed to q(x t+1 ) as a query and H t It is a multi-head attention mechanism with the following key-values.

[0069]

number

[0070]

number

[0071] The experimental model outputs y t+1 Next, x t+1 Using the following formula (8), we can obtain the quantity R consisting of N tokens. t+1 After obtaining the hidden state H t+1 is obtained by the following equation (9).

[0072]

number

[0073]

number

[0074] In the experimental model, MaxPool(·) was used as Update(·) in equation (1), which is a Max-Pooling function that selects the maximum value from the elements in the channel direction.

[0075] More specifically, this experimental model was a DNN (Deep Neural Network) architecture that integrated a Flashback mechanism into both Transformers and Mamba. The Flashback mechanism is a memory mechanism with Flashback properties. The Flashback property is a property that allows the experimental model to avoid the memory degradation and gradient vanishing or explosion described above. Therefore, the Flashback mechanism is specifically a hidden vector H t is the memory, and H t This is a process of determining the n-th element of the N global tokens that make up the global token ∂ ...

[0076] The Transformer++ described in Reference 1 below was used as the basis for the Transformers architecture in such a DNN architecture.

[0077] Reference 1: Touvron, H., Martin, L., Stone, K. et al.: Llama 2:Open Foundation and Fine-Tuned Chat Models, arXiv preprint arXiv:2307.09288 (2023).

[0078] In the Transformers architecture, an attention mechanism including an FFN (Feed-Forward Network) exists as a set of blocks. In Mamba, an SSM (Selective State Space Model) architecture integrated with MLP (Multi-layer Perceptron) processing exists as a set of blocks. In addition, a flashback mechanism is inserted into the even-numbered block positions of Transformer++ and Mamba. This allows the flashback mechanism to be executed in the attention mechanism and RNN processing. As a result, the vector x t is obtained by the sliding window attention mentioned above. Also, the vector x t is obtained by a recurrent neural network. Furthermore, the hidden vector h t+1 From data z t+1 Also, the recurrent neural network generates the hidden vector h t+1 From data z t+1 The Flashback feature was introduced into the architecture by inserting a Flashback mechanism into the even-numbered block positions of Transformer++ and Mamba. At this time, for the Transformers architecture, the Sliding Window Attention described in Reference 2 below was used to keep the processing efficiency constant relative to the input sequence length.

[0079] Reference 2: Eltagy, I., Peters, ME and Cohan, A.: Long former: The Long-Document Transformer, arXiv preprint arXiv:2004.05150 (2020).

[0080] The embedding dimensions of the input and output tokens in the Flashback mechanism were preset to match the architectures of Transformer++ and Mamba.

[0081] In experiments, the effectiveness of the experimental model in the language modeling task of next-token prediction was verified using the Pile dataset. The Pile dataset consists of 22 high-quality subsets with a total capacity of 800 GB. These subsets include papers, books, programs, technical documents, etc. Therefore, the experiments evaluated the generalization of the language model using a wide variety of text data.

[0082] In the experiment, the BASED-based technology described in the following reference 3 was used as the comparison technology. All of the comparison technologies were based on a model size of about 360 megabytes of parameters.

[0083] Reference 3: Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Alberti, S., Zinsley, D., Zou, J., Rudra, A. and R´e, C.: Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff, ICML (2024).

[0084] In both the experimental model and the comparison technology, the batch size was 8, the embedding dimension of the input and output of each block was 1024, and the input sequence length was 2048. In the experimental model, the number of heads was 16, and FlashAttention-2 was used for calculating the attention mechanism. The window size of the sliding window attention in the experimental model was 512, and the number of global tokens was N=1.

[0085] The accuracy of language modeling was evaluated using perplexity (Ppl.) for token prediction. The processing speed of the experimental model and the comparison technologies was compared using the token generation throughput per unit time (Tok. / ms). The sequence length for each technology was fixed at 2048. To fairly compare the performance of the experimental model and the comparison technologies, the same GPU was used for the evaluation. The patch size for each technology was set to maximize throughput within the GPU's executable range. The memory efficiency of each technology was compared using the total GPU memory usage during processing speed measurement divided by the batch size (MiB / Seq.). Furthermore, the total number of blocks in the experimental model was set to match the token prediction accuracy described below.

[0086] FIG. 6 is a diagram illustrating an example of experimental results in an embodiment. More specifically, FIG. 6 shows the results of comparing the token prediction accuracy, processing speed, and memory usage of each language modeling technology (i.e., the experimental model and the comparative technology). In FIG. 6, in the "w / Flashback" column, a check mark indicates that the Flashback mechanism was inserted at even-numbered block positions in Transformer++ and Mamba, and no check mark indicates that the Flashback mechanism was not inserted. Note that a check mark is marked for a technology whose architecture is Transformers and whose method is Transformers++ described in Reference 1. Furthermore, a check mark is not marked for a technology whose architecture is a recurrent neural network and whose method is Based described in Reference 3. A check mark is marked for a technology whose architecture is a recurrent neural network and whose method is Mamba described in Reference 4 below.

[0087] Reference 4: Gu, A. and Dao, T.: Mamba: Linear-Time Sequence Modeling with Selective State Spaces, arXiv preprint arXiv:2312.00752 (2023).

[0088] Figure 6 shows that introducing the Flashback mechanism into Transformer++ results in a speedup of more than three times and a 74% reduction in memory usage. Figure 6 also shows that, even though the input sequence length at the time of evaluation was 2048, which exceeds the window size of 512 in Sliding Window Attention, the token prediction accuracy of Transformer++ improved by -0.01 Ppl. Figure 6 thus demonstrates the effectiveness of the Flashback mechanism as a memory mechanism in the Transformers architecture.

[0089] Figure 6 shows that by introducing the Flashback mechanism into Mamba, we achieved a -0.01 Ppl improvement in accuracy, a 29% reduction in memory usage, and a 5% increase in speed. Figure 6 also shows that Mamba with the Flashback mechanism achieved a -0.01 Ppl improvement in accuracy, a 55% reduction in memory usage, and a 2.8x increase in speed compared to BASED, a recurrent neural network-based State-of-the-Art method.

[0090] The learning device 1 configured in this manner includes a control unit 11 that executes the learning process. Therefore, as described in <Effects of the learning process>, the learning device 1 can suppress deterioration in the accuracy of the series processing.

[0091] Furthermore, the estimation device 2 configured in this manner performs estimation using the learning results obtained by the learning device 1. Therefore, the estimation device 2 can suppress deterioration in accuracy of series processing.

[0092] Furthermore, the information processing system 100 configured in this manner includes the learning device 1. Therefore, the information processing system 100 can suppress deterioration in accuracy of series processing.

[0093] (Variation) It should be noted that, if the following first, second and third vector conditions are satisfied, the element rule may be, for example, a category-dependent rule.

[0094] The first vector condition is the feature vector Q t The condition is that each element of h belongs to one of M categories (M is a predetermined integer between 1 and D). The second vector condition is that the hidden vector h t-1 The condition is that each element of belongs to one of M categories.

[0095] The third vector condition is that the first element d belongs to a category with a feature vector Q t In addition to the d-th element of the feature vector Q t If the d'th element (d' is an integer between 1 and D and is different from d) belongs to the category of the second-type element d, the hidden vector h t-1 The condition is that the d'th element of also belongs to the set.

[0096] The category of the first element d is the feature vector Q t The category to which the d-th element of the second kind of element d belongs is the hidden vector h t-1 is the category to which the dth element of belongs.

[0097] The category-dependent rules are defined as hidden vectors h t A category-dependent rule is a rule that includes a first subrule, a second subrule, and a third subrule.

[0098] The first sub-rule states that if the maximum value of the elements belonging to the category of the first-type element d´´ is greater than the maximum value of the elements belonging to the category of the second-type element d´´, then the hidden vector h tThe rule is that the value of the element of d' is the first value. The first value is the feature vector Q that belongs to the category of the first element d'. t is the value of the d´´th element of

[0099] The second sub-rule states that if the maximum value of the elements belonging to the category of the first-type element d´´ is smaller than the maximum value of the elements belonging to the category of the second-type element d´´, then the hidden vector h t The rule is that the value of the element of d' is the second. The second value is the hidden vector h that belongs to the category of the second element d'. t-1 is the value of the d´´th element of

[0100] The third sub-rule states that if the maximum value of the elements belonging to the category of the first-type element d´´ is the same as the maximum value of the elements belonging to the category of the second-type element d´´, then the hidden vector h t The rule is that the value of an element of d' is set to one of the first value and the second value according to a predetermined rule.

[0101] FIG. 7 shows the hidden vector h obtained according to the category-dependent rules in the modified example. t 7 is a diagram showing an example of a hidden vector h t-1 The first and second elements of belong to the same category C201, and the hidden vector h t-1 The third and fourth elements of the feature vector Q belong to the same category C202. t-1 The first and second elements of belong to the same category C201, and the feature vector Q t-1 The third and fourth elements belong to the same category C202.

[0102] Hidden vector h t-1 In the category C101 including the first and second elements of the hidden vector h t-1In category C102, which includes the third and fourth elements, the value of the third element is the maximum value.

[0103] Feature vector Q t In the category C201 including the first and second elements of the feature vector Q, the value of the first element is the maximum value. t In the category C202 that includes the third and fourth elements, the value of the fourth element is the maximum value.

[0104] Therefore, in the example of Figure 7, the hidden vector h t The value of the first element of the hidden vector h t-1 In the example in Figure 7, the hidden vector h t The value of the second element of the hidden vector h t-1 In the example in Figure 7, the hidden vector h t The value of the third element of the vector x t In the example in Figure 7, the hidden vector h t The value of the fourth element of the vector x t is the value of the fourth element of

[0105] The learning device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.

[0106] The estimation device 2 may be implemented using a plurality of information processing devices communicably connected via a network. In this case, the processes executed by the control unit 21 may be distributed among the plurality of information processing devices.

[0107] All or part of the functions of the information processing system 100, the learning device 1, and the estimation device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0108] The control unit 21 is an example of an estimation unit.

[0109] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0110] 100...information processing system, 1...learning device, 2...estimation device, 11...control unit, 12...interface unit, 13...storage unit, 21...control unit, 22...interface unit, 23...storage unit, 91...processor, 92...memory, 93...processor, 94...memory

Claims

1. The quantity at time t, which is the tth time (t is an integer equal to or greater than 1), is a D-dimensional vector x ｔ a control unit that performs learning of a learning object model, which is a mathematical model that estimates second time series data that is the same as or different from the first time series data, based on first time series data where D is an integer equal to or greater than 1; Equipped with The learning target model is a vector x ｔ and the hidden vector h ｔ－１ The hidden vector h obtained based on ｔ and estimating data at time t in the second time series data based on the The hidden vector h ｔ is a D-dimensional vector, and the vector x ｔ The value of the dth element (d is an integer between 1 and D) in the D-dimensional feature vector of ｔ－１ and the value of the d-th element in The hidden vector h at the first time １ is the vector x １ is a D-dimensional feature vector of Learning device.

2. The predetermined rule is that the vector x ｔ The value of the dth element in the D-dimensional feature vector and the hidden vector h ｔ－１ The value of the d-th element in ｔ The rule is that the value of the d-th element of The learning device according to claim 1 .

3. The predetermined rule is that the vector x ｔ The value of the dth element in the D-dimensional feature vector and the hidden vector h ｔ－１ The value of the d-th element in ｔ The rule is that the value of the d-th element of The learning device according to claim 1 .

4. The mathematical model is represented by a neural network. The learning device according to claim 1 .

5. The vector x ｔ Each element of the D-dimensional feature vector belongs to one of M categories (M is a predetermined integer between 1 and D), The hidden vector h ｔ－１ Each element of belongs to one of the M categories, The vector x ｔ If a d'th element (d' is an integer between 1 and D and is different from d) belongs to the category of the first type element d, which is the category to which the d'th element of the D-dimensional feature vector belongs, other than the d'th element, then the hidden vector h ｔ－１ The hidden vector h ｔ－１ The d'th element of also belongs to The predetermined rule is: Hidden vector h ｔ The value of the element d' (d' is an integer between 1 and D) of If the maximum value of the elements belonging to the category to which the first-type element d'' belongs is greater than the maximum value of the elements belonging to the category to which the second-type element d'' belongs, the vector x ｔ Let the value of the d'th element of the D-dimensional feature vector be If the maximum value of the elements belonging to the category to which the first-type element d'' belongs is smaller than the maximum value of the elements belonging to the category to which the second-type element d'' belongs, the hidden vector h ｔ－１ Let the value of the d´´th element of If the maximum value of the elements belonging to the category to which the first-type element d'' belongs is the same as the maximum value of the elements belonging to the category to which the second-type element d'' belongs, the vector x ｔ The value of the d'th element of the D-dimensional feature vector and the hidden vector h belonging to the category of the second type element d' ｔ－１ The value of the d'th element of The rule is, The learning device according to claim 1 .

6. The vector x ｔ is a time series data of a D-dimensional vector, and the vector at each time t is the vector x ｔ The vector obtained by the Sliding Window Attention mechanism based on the original data, which may be the same or different time series data as the original data. The learning device according to claim 1 .

7. The mathematical model uses a sliding window attention mechanism to find the hidden vector h （ｔ＋１） get output from, The learning device according to claim 1 .

8. The vector x ｔ is a time series data of a D-dimensional vector, and the vector at each time t is the vector x ｔ A vector obtained by a recurrent neural network based on the original data, which may be the same or different time series data as The learning device according to claim 1 .

9. The mathematical model obtains output from a hidden vector h(t+1) using a recurrent neural network. The learning device according to claim 1 .

10. The learning of the mathematical model is self-supervised learning. The learning device according to claim 1 .

11. The learning uses backpropagation. The learning device according to claim 1 .

12. The quantity at time t, which is the tth time (t is an integer equal to or greater than 1), is a D-dimensional vector x ｔ (D is an integer of 1 or more), and a control unit that performs learning of a learning object model that is a mathematical model that estimates second time series data that is the same as or different from the first time series data, based on the first time series data, ｔ and the hidden vector h ｔ－１ The hidden vector h obtained based on ｔ and estimating the data at time t in the second time series data based on the hidden vector h ｔ is a D-dimensional vector, and the vector x ｔ The value of the dth element (d is an integer between 1 and D) in the D-dimensional feature vector of ｔ－１ and the value of the d-th element in １ is the vector x １ an estimation unit that performs estimation using the results obtained by the learning device, which is a D-dimensional feature vector of An estimation device comprising:

13. The quantity at time t, which is the tth time (t is an integer equal to or greater than 1), is a D-dimensional vector x ｔ (D is an integer of 1 or more), and a control unit that performs learning of a learning object model that is a mathematical model that estimates second time series data that is the same as or different from the first time series data, based on the first time series data, ｔ and the hidden vector h ｔ－１ The hidden vector h obtained based on ｔ and estimating the data at time t in the second time series data based on the hidden vector h ｔ is a D-dimensional vector, and the vector x ｔ The value of the dth element (d is an integer between 1 and D) in the D-dimensional feature vector of ｔ－１ and the value of the d-th element in １ is the vector x １ A learning method executed by a learning device, in which the vector is a D-dimensional feature vector of a learning step in which the control unit performs the learning; A learning method that has

14. The quantity at time t, which is the tth time (t is an integer equal to or greater than 1), is a D-dimensional vector x ｔ (D is an integer of 1 or more), and a control unit that performs learning of a learning object model that is a mathematical model that estimates second time series data that is the same as or different from the first time series data, based on the first time series data, ｔ and the hidden vector h ｔ－１ The hidden vector h obtained based on ｔ and estimating the data at time t in the second time series data based on the hidden vector h ｔ is a D-dimensional vector, and the vector x ｔ The value of the dth element (d is an integer between 1 and D) in the D-dimensional feature vector of ｔ－１ and the value of the d-th element in １ is the vector x １ an estimation method executed by an estimation device including an estimation unit that performs estimation using a result obtained by a learning device, the result being a D-dimensional feature vector of an estimation step in which the estimation unit performs the estimation; An estimation method having:

15. A program for causing a computer to function as the learning device according to any one of claims 1 to 11.

16. A program for causing a computer to function as the estimation device according to claim 12.

Images