Series data processing device and program

Abstract

To provide a series data processing device and a program that can consistently learn rearward series and prevent calculation efficiency from degrading even if a predicted length is increased.SOLUTION: The series data processing device includes a coefficient estimation unit. The coefficient estimation unit is configured to include a model for, based on previous context (X<k) at a position k of first series data, estimating an estimate value (C'k) of a coefficient of an approximate function that approximates a value t(X≥k) after the position k of the first series data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a sequence data processing device and a program. [Background technology]

[0002] [Discrete autoregressive model] An autoregressive model (reference URL: https: / / ja.wikipedia.org / wiki / %E8%87%AA%E5%B7%B1%E5%9B%9E%E5%B8%B0%E3%83%A2%E3%83%87%E3%83%AB) is a stochastic process in which the model output at time k depends on the model output before time k (hereafter called the previous context), and is usually defined in discrete time. The model output X at time k in a linear autoregressive model of order p (depending on the p model outputs immediately preceding time k) is k (scalar) can be formulated as equation (1) below.

[0003]

number

[0004] In the formula (1), φ i and c are parameters, and ε k is the error (noise) term at time t. The linear autoregressive model expressed by equation (1) can be naturally extended to the case where the output is multivariate (vector), and is generally called a vector autoregressive model. In addition, similar conditioning (i.e., X k is the previous context value X <k =X k-1 ,X k-2 We can also consider a nonlinear (vector) model where the vectors depend on the eigenvalues ​​(depending on ,···), which is commonly called a nonlinear (vector) autoregressive generative model.

[0005] In recent machine learning fields, an autoregressive model usually refers to a nonlinear vector autoregressive generative model. This nonlinear vector autoregressive generative model can be configured with a recurrent neural network such as LSTM (Long Short-term Memory) or a deep neural network such as Transformer, and parameters can be learned using discrete-time time series learning data. Non-Patent Document 1 describes LSTM. Non-Patent Document 2 describes Transformer.

[0006] Learning of an autoregressive model is realized by inputting each value (vector) of the discrete-time prior context to a (deep) neural network configured as described above, and adjusting the network parameters using stochastic gradient descent or the like to minimize the error between the predicted value (vector) that is the network output and the actual value (vector) at the current time. When the time series data is continuous and the model is trained (regression training) to take values ​​as close as possible to the training data, mean square error or the like is used for the error. On the other hand, when the time series data is symbols and the model is trained (classification training) to increase the probability of predicting the symbols of the training data as much as possible, the network output is the output probability for each symbol set that is the output candidate, and cross entropy error or the like is used for the error. In particular, in this example, when the training data is text and each data value is a word, etc., such a model is called a language model.

[0007] [Continuous-time autoregressive model] The discrete-time autoregressive model described above can, in principle, be naturally extended to continuous time. This is called the Ornstein-Uhlenbeck process (Reference URL: https: / / ja.wikipedia.org / wiki / Ornstein_Uhlenbeck_process_2015). The Ornstein-Uhlenbeck process is generally formulated as in equation (2) below. That is, if the output r at a certain time t is the previous output r <t and is defined by the stochastic differential equation (2).

[0008]

number

[0009] In equation (2), θ, μ, and σ are parameters. t denotes the Wiener process. Any discrete solution of the stochastic differential equation in (2) is a formulation of the discrete-time autoregressive model.

[0010] When considering an autoregressive model, the difference between whether some time series data is essentially defined in discrete time or essentially defined in continuous time is not important. Time series data essentially defined in discrete time has no concept of sampling rate. Discrete time series data essentially defined in continuous time is, for example, an observation value sequence observed at a specified sampling rate, such as an audio waveform, which is obtained as a result of observation as values ​​corresponding to discrete times. Both of these types of time series data can be modeled using the discrete-time autoregressive model described above.

[0011] However, when observing the change in a value defined in continuous time with a sampling rate (relatively frequent sampling) that is large compared to the change, the observed value at time t is often almost the same as the observed value at the previous time. In such cases, it is difficult to set a meaningful prediction problem as an autoregressive task. This problem can generally be alleviated by making the sampling rate smaller, but in that case, details of the data are lost.

[0012] [Applications of autoregressive models] The prediction ability of sequence modeling using an autoregressive model can be directly utilized, for example, in speech recognition to narrow down next word candidates using a language model. In addition, the ability to interpret the previous context obtained as a result of improving the prediction performance can be used to solve various other tasks. An autoregressive model that has been trained to predict a subsequent waveform from the previous context of a speech waveform is likely to have grasped the characteristics of the previous context to the extent that it can predict how the subsequent waveform will develop. Therefore, it may be possible to use this trained autoregressive model to solve some other task according to the characteristics of the previous context with higher performance. Prediction model training performed for this purpose is generally called "pre-training using self-supervised learning". Autoregressive model training is suitable for such applications.

[0013] [Autoregressive model for long-term prediction] As described above in the explanation of the continuous-time autoregressive model, in order to set a meaningful prediction problem in the learning of an autoregressive model, it may not be sufficient to predict only one current time from a previous context. In addition, for example, in the field of reinforcement learning for autonomous robots, the usefulness of directly predicting a value at a distant future time or predicting several consecutive values ​​from the current time has been asserted in order to acquire the ability to make a decision to initiate a time-consuming action in preparation for an event that will occur in the future distant from the current time. Non-Patent Document 3 describes a technology for directly predicting a value at a distant future time. Non-Patent Document 4 describes a technology for predicting multiple consecutive values ​​from the current time.

[0014] In the context of the above-mentioned prior learning, learning to predict a longer sequence from the previous context may lead to further improvement of the ability to interpret the previous context. Although models that predict sequences beyond the current time are not strictly defined as autoregressive models, they will be referred to as autoregressive models in this patent specification.

[0015] Realizing an autoregressive model that can make long-term predictions as described above will help expand the scope of application of autoregressive models and is expected to improve their performance as pre-training models, thereby increasing their usefulness. [Prior art documents] [Non-patent literature]

[0016] [Non-Patent Document 1] Sepp Hochreiter, Jurgen Schmidhuber, Long Short-term Memory, Neural Computation, 9(8):1735-80, 1997. [Non-Patent Document 2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, Attention Is All You Need, In proceeding of Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017, https: / / proceedings.neurips.cc / paper / 2017 / file / 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf [Non-Patent Document 3] Karol Gregor,George Papamakarios,Frederic Besse,Lars Buesing,Theophane Weber,Temporal Difference Variational Auto-Encoder,In proceeding of International Conference on Learning Representations (ICLR 2019),2019,https: / / openreview.net / pdf?id=S1x4ghC9tQ [Non-Patent Document 4] Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, Aaron van den Oord, Shaping Belief States with Generative Environment Models for RL, In proceeding of Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019, https: / / proceedings.neurips.cc / paper / 2019 / file / 2c048d74b3410237704eb7f93a10c9d7-Paper.pdf Summary of the Invention [Problem to be solved by the invention]

[0017] However, there are problems with the background technologies described above. For example, the technology described in Non-Patent Document 3 has a problem that it is not possible to learn the consistency of the future because it predicts the future at one point. Also, the technology described in Non-Patent Document 4 has a problem that it is necessary to generate the desired future sequence gradually. In other words, in the technology of Non-Patent Document 4, during learning, the previous context X <k From, the output value X k ,X k+1 , ... <k+1 to output value Xk+1 ,X k+2 ,..., and calculate the error. Therefore, if the prediction length is increased, the amount of calculation increases, resulting in poor calculation efficiency.

[0018] The present invention has been made based on the above-mentioned problem recognition, and aims to provide a sequence data processing device and a program that can consistently learn future (backward) sequences and does not deteriorate in calculation efficiency even when the prediction length becomes long. [Means for solving the problem]

[0019] [1] In order to solve the above problem, a sequence data processing device according to one aspect of the present invention is a sequence data processing device that calculates a previous context (X <k ), the value (X ≧k ) is the estimated coefficient of the approximation function (C´ k a coefficient estimator comprising a model for estimating

[0020] [2] In addition, one aspect of the present invention is a sequence data processing device according to the above [1], further comprising: ≧k ) based on the correct answer, the value (X ≧k ) is the coefficient of the approximation function (C k ), and the coefficient (C k ) to obtain the value (X ≧k A sequence value estimator having a model for calculating at least a part of the estimated value of the coefficient (C' k ) and the coefficient (C k ) and the coefficient error (LC k ), and the value (X ≧k ) and the value after position k of the first sequence data (X ≧k) and the series value error (LV k ) calculated by the coefficient error calculation unit, k ) and the sequence value error (LV k ) based on which the values ​​of the learnable internal parameters of the model provided in the coefficient estimator and the values ​​of the learnable internal parameters of the model provided in the sequence value estimator can be adjusted.

[0021] [3] In addition, according to one aspect of the present invention, in the sequence data processing device according to the above [1], the coefficient estimating unit is <k ), and further based on a predetermined value (y k-1 ), and the coefficient estimation unit calculates the predetermined value (y k-1 Based on the second sequence data, which is a sequence of y ≧k ), the value (X ≧k ) is the coefficient of the approximation function (C k ), and the coefficient (C k ) to obtain the value (X ≧k A sequence value estimator having a model for calculating at least a part of the estimated value of the coefficient (C' k ) and the coefficient (C k ) and the coefficient error (LC k ), and the value (X ≧k ) and the value (y ≧k ) and the series value error (LV k ) calculated by the coefficient error calculation unit, k ) and the sequence value error (LV k) based on which the values ​​of the learnable internal parameters of the model provided in the coefficient estimator and the values ​​of the learnable internal parameters of the model provided in the sequence value estimator can be adjusted.

[0022] [4] In addition, the sequence data processing device according to one aspect of the present invention calculates a previous context (X <k ), the value (X ≧k ) is the coefficient of the approximation function (C k The present invention also includes a distribution statistical information estimation unit having a model for calculating a distribution statistical information estimate, which is an estimate of distribution statistical information, which is statistical information regarding the distribution of the vectors.

[0023] [5] In addition, one aspect of the present invention is a sequence data processing device according to the above [4], further comprising: ≧k ) based on the correct answer, the value (X ≧k ) is the coefficient of the approximation function (C k ) of the approximation function calculated by the coefficient calculation unit; k Based on 1≦k≦L, the coefficient (C k A distribution statistical information calculation unit having a model for obtaining distribution statistical information, which is statistical information regarding the distribution of k (1≦k≦L), and a sample (Z k ), for a given k (1≦k≦L) of the first sequence data, the k-th coefficient (C k ) estimate (C´ k ) and the k-th coefficient (C k ) based on the k-th value of the first sequence data, the estimate (X´ k ) and the k-th coefficient (C k ) estimate (C´ k ) and the k-th coefficient (C k) and the coefficient error (LC k ) a coefficient error calculation unit; The predicted value (X' k ) and the kth value of the given sequence data (X k ) and the series value error (LV k ) which is an error between the distribution statistical information estimated value calculated by the distribution statistical information estimation unit and the distribution statistical information calculated by the distribution statistical information calculation unit, k ), and configured to adjust, based on the coefficient error, the sequence value error, and the distribution statistical information error, values ​​of the learnable internal parameters of the model provided in the distribution statistical information calculation unit, the values ​​of the learnable internal parameters of the model provided in the distribution statistical information estimation unit, the values ​​of the learnable internal parameters of the model provided in the coefficient estimation unit, and the values ​​of the learnable internal parameters of the model provided in the sequence value estimation unit.

[0024] [6] In addition, in one aspect of the present invention, in the sequence data processing device according to [4], the distribution statistical information estimation unit is <k ), and further based on a predetermined value (y k-1 ), and the distribution statistical information estimation unit calculates the predetermined value (y k-1 Based on the second sequence data, which is a sequence of y ≧k ), the value (X ≧k ) is the coefficient of the approximation function (C k ) of the approximation function calculated by the coefficient calculation unit; k Based on 1≦k≦L, the coefficient (C k A distribution statistical information calculation unit having a model for obtaining distribution statistical information, which is statistical information regarding the distribution of k (1≦k≦L), and a sample (Z k), for a given k (1≦k≦L) of the first sequence data, the k-th coefficient (C k ) estimate (C´ k ) and the k-th coefficient (C k ) based on the k-th value of the first sequence data, the estimate (X´ k ) and the k-th coefficient (C k ) estimate (C´ k ) and the k-th coefficient (C k ) and the coefficient error (LC k a coefficient error calculation unit that calculates a predicted value (X' k ) and the k-th value of the second sequence data (y k ) and the series value error (LV k ) which is an error between the distribution statistical information estimated value calculated by the distribution statistical information estimation unit and the distribution statistical information calculated by the distribution statistical information calculation unit, k ), and configured to adjust, based on the coefficient error, the sequence value error, and the distribution statistical information error, values ​​of the learnable internal parameters of the model provided in the distribution statistical information calculation unit, the values ​​of the learnable internal parameters of the model provided in the distribution statistical information estimation unit, the values ​​of the learnable internal parameters of the model provided in the coefficient estimation unit, and the values ​​of the learnable internal parameters of the model provided in the sequence value estimation unit.

[0025] [7] Another aspect of the present invention is a program for causing a computer to function as the sequence data processing device according to any one of [1] to [6] above. Effect of the Invention

[0026] According to the present invention, the trained sequence data processing device can process sequence data (for example, estimate sequence data or estimate coefficients for calculating sequence data). Furthermore, the sequence data processing device according to one aspect of the present invention can learn sequence data. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a schematic diagram for conceptually explaining coefficient errors in each embodiment of the present invention. [Diagram 2] 1 is a block diagram showing a schematic functional configuration of a sequence data processing device according to a first embodiment of the present invention. [Diagram 3] 4 is a flowchart showing a procedure for performing machine learning of learnable parameters in the sequence data processing device according to the first embodiment. [Figure 4] FIG. 11 is a block diagram showing a schematic functional configuration of a sequence data processing device according to a second embodiment. [Diagram 5] 13 is a flowchart showing a procedure for performing machine learning of learnable parameters in the sequence data processing device according to the second embodiment. [Figure 6] FIG. 11 is a block diagram showing a schematic functional configuration of a sequence data processing device according to a third embodiment. [Figure 7] 13 is a flowchart showing a procedure for performing machine learning of learnable parameters in the sequence data processing device according to the third embodiment. [Figure 8] FIG. 13 is a block diagram showing a schematic functional configuration of a series data processing device according to a modified example of the embodiment. [Figure 9] FIG. 1 is a block diagram showing an example of the configuration of a computer for implementing the first, second, and third embodiments or modifications thereof. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Next, several embodiments of the present invention will be described with reference to the drawings. In the following description, LSTM is an abbreviation for Long Short-term Memory. Furthermore, RNN is an abbreviation for Recurrent Neural Network. The sequence data processing device (1, 2, 3) of each embodiment processes sequence data. The sequence data is data ordered as..., (k-1)th, kth, (k+1)th,.... Each piece of sequence data belongs to a predetermined sequence data set and has characteristics specific to that sequence data set. The sequence data processing device (1, 2, 3) according to each of the following embodiments utilizes those characteristics.

[0029] In each embodiment described below, a recurrent neural network HiPPO is used. HiPPO is a neural network that has the ability to aggregate an infinitely long sequence into a vector of a fixed size in principle. HiPPO is described in [References: Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, Christopher Re, HiPPO: Recurrent Memory with Optimal Polynomial Projections, In proceeding of Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020, https: / / proceedings.neurips.cc / paper / 2020 / file / 102f0bb6efb3a6128a3c750dd16729be-Paper.pdf]. In the embodiment described below, a future sequence aggregate vector at each time is calculated at high speed using HiPPO, and then the prediction problem in the autoregressive model learning is changed so that the future sequence aggregate vector at each time is estimated from the previous context at the time. This solves the problems of the conventional technology.

[0030] The configuration (special technical configuration) common to each embodiment (including modified examples) described below is as follows. That is, in the special technical configuration of the invention, the coefficient calculation units 1113, 2113, 3113, and 4113 (HiPPO RNN processing units) calculate the given sequence data (X k where k is an integer that represents the order of the sequence, and the coefficient (C k ; 1≦k≦L). Specifically, the coefficient calculation units 1113, 2113, 3113, and 4113 (HiPPO RNN processing units) calculate X ≧k (X k ,X k+1 ,···,X L ) based on the coefficient C k Calculate the series data X 1 ,X 2 ,···,X L When the coefficient calculation units 1113, 2113, 3113, and 4113 (HiPPO RNN processing units) are given, the coefficient calculation units 1113, 2113, 3113, and 4113 (HiPPO RNN processing units) calculate the sequence in reverse order, that is, C L ,C L-1 ,···,C 2 ,C 1 Here, the coefficient calculation units 2113 and 4113 can calculate these coefficients in the order of the sequence data X 1 ,X 2 ,···,X L Instead of this, another sequence data y 0 ,y 1 ,y 2 ,···,y L The coefficient estimation units 1111 and 2111 may calculate the coefficients based on the preceding context (X <k ) based on the coefficients (C k ) estimate (C´ k Alternatively, the coefficient estimation units 3164 and 4164 estimate the coefficients (C k ) estimate (C´ kThe sequence value estimation units 1112, 2112, 3112, and 4112 estimate the coefficients (C k ) to estimate the k-th value of the series data (X´ k The coefficient error calculation units 1114, 2114, 3114, and 4114 calculate the coefficients (C k ) estimate (C´ k ) and the coefficients of the approximation functions calculated by the coefficient calculation units 1113, 2113, 3113, and 4113 (HiPPO RNN processing units) (C k The sequence value error calculation units 1115, 2115, 3115, and 4115 calculate the coefficient error, which is the error between the estimated values ​​(X´ k ) and the value (X k ) and a sequence value error, which is the error between them. Furthermore, the overall error calculation units 1116, 2116, 3116, and 4116 calculate an overall error based on at least the coefficient error and the sequence value error. Furthermore, the control units 1110, 2110, 3110, and 4110 perform control to adjust at least the values ​​of internal parameters of the models held by the coefficient estimating units 1111, 2111, 3164, and 4164 and the values ​​of internal parameters of the models held by the sequence value estimating units 1112, 2112, 3112, and 4112, respectively, based on the overall error.

[0031] FIG. 1 is a schematic diagram conceptually illustrating the above-mentioned coefficient error. As shown in the figure, for each k=1, . . . , L, <k Based on this, C´ k is estimated. That is, X 1 Based on C´ 2 is estimated, and X 1 ,X 2 Based on C´ 3 is estimated, and so on, until X 1 ,X 2 ,···,XL-1 Based on C´ L is estimated. On the other hand, for each k = 1, ···, L, based on X ≧k C k is calculated. That is, based on X L C L is calculated, and based on X L-1 , X L C L-1 is calculated, and it proceeds similarly hereinafter. Based on X 1 , X 2 , ···, X L C 1 is calculated. The coefficient error is the error between C´ k and C k . In the learning in the embodiment, for example, the series data processing device optimizes the model having the sum of the errors between C´ k and C k for each k = 1, ···, L as the coefficient error.

[0032] Hereinafter, a plurality of specific embodiments and modifications will be described.

[0033] [First Embodiment] First, the HiPPO used in this embodiment will be described. The HiPPO method described in the above reference of Gu et al. is as follows.

[0034] First, regarding the problem of regarding a (discrete) time-series scalar input as a sampling result of some continuous function and storing the time-series input from time 0 to s, it is defined as follows. That is, this problem is defined as "for a continuous function f(t) defined on -∞ < t < ∞, considering an approximate function g(t; s) that is closest to the function f(t) by some measure based on the sampling results at times t = 0, ···, s, and holding the coefficients c(s) of the orthonormal basis truncated at a finite order of g(t; s)".

[0035] Next, in the above setting, it is known that the coefficient c(s) of the orthonormal basis at time t satisfies the ordinary differential equation of the following formula (3).

[0036]

Number

[0037] In some combinations of orthonormal bases and measure functions, A(s) is known to be a closed-form constant matrix that does not contain s (this is called a HiPPO matrix). For example, when selecting the (normalized) Legendre polynomial basis truncated at the N-th order as the orthonormal basis and ω(t;s)=e -(s-t) (-∞ < t ≤ s) as the measure function, A(s) is an N×N constant matrix that does not contain s, and its value is as shown in Equation (4) below. In the above reference, this N×N constant matrix is called HiPPO-LegS.

[0038]

Number

[0039] However, A nm is the element in the n-th row and m-th column of the matrix A(s). Also, B(s) in Equation (3) is an N-dimensional vector. In the above reference, by setting the value of B(s) to a constant vector that does not contain s represented by Equation (5) below, it is assumed that g(t;s) is an approximation of a continuous function having the input sequence as samples.

[0040]

Number

[0041] However, B n in Equation (5) is the n-th dimensional element of the vector B(s). Hereafter, A and B are described as not containing s (time-invariant), but in principle, they may be time-varying.

[0042] In the above reference, A and B for combinations of orthonormal bases and velocity functions other than this HiPPO-LegS are also described. That is, A and B are not limited to the above.

[0043] However, when using the above HiPPO-LegS matrix, the approximation function corresponding to the coefficients obtained by the above ordinary differential equation has the following characteristics. That is, the closer t is to s, the more accurate the approximation is, but the more distant t is from s, the more the accuracy of the approximation is lost exponentially. In discrete terms, the above approximation function has the measure property that it remembers the details of the most recent input, but remembers only the outline of distant inputs (inputs that are past in the chronological order). Therefore, it is preferable for summarizing and storing an infinitely long input sequence.

[0044] By discretizing the ordinary differential equation above, it is possible to obtain c(s) at discrete times as a recurrence formula. Any method can be used as the discretization method. For example, if the (forward) Euler method is used as the discretization method, and some Δ (this Δ corresponds to the discretization step width) is prepared, it can be expressed as shown in the following equations (6) and (7).

[0045]

number

[0046]

number

[0047] In addition, f in equation (7) k is the kth input in the discrete sequence. Also, c k is the coefficient at the time of the k-th input. The calculation process shown in equations (6) and (7) can be calculated as a (discrete) linear RNN, and as mentioned above, it is an embodiment of an RNN that has the ability to properly memorize long sequences.

[0048] The degree N of the orthonormal basis is related to memory capacity. Also, Δ is related to the proportion of memory capacity consumed to store each discrete input. If Δ is reduced while keeping N fixed, a longer sequence can be stored, but the accuracy of each input is lost. On the other hand, if Δ is increased while keeping Δ fixed, each input can be stored more accurately while maintaining the length of the memory.

[0049] In the above description of HiPPO, the input and output values ​​are scalars, but the input and output can be vectors by configuring the above RNN in parallel for the required number of dimensions. In other words, the above described method can be generalized as a configuration in which the above RNNs are arranged in parallel for the required number of dimensions (for example, the number of dimensions is D) as input and output values ​​are vectors, including the case where the number of dimensions is 1 (in this case the value is a scalar).

[0050] [Configuration of the sequence data processing device of the first embodiment] FIG. 2 is a block diagram showing a schematic functional configuration of the sequence data processing device according to the present embodiment. As shown in the figure, the sequence data processing device 1 includes a control unit 1110, a coefficient estimation unit 1111, a sequence value estimation unit 1112, a coefficient calculation unit 1113, a coefficient error calculation unit 1114, a sequence value error calculation unit 1115, and a total error calculation unit 1116. Each of these functional units can be realized, for example, by a computer and a program. Each functional unit has a storage unit as necessary. The storage unit is, for example, a variable in the program or a memory allocated by the execution of the program. Non-volatile storage units such as a magnetic hard disk drive or a solid state drive (SSD) may be used as necessary. At least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

[0051] The sequence data processing device 1 is configured as a device capable of machine learning. The mechanism of machine learning itself can be realized by using existing technology. At least some of the functions of the sequence data processing device 1 have learnable parameters. The values ​​of these parameters are stored, for example, in a rewritable memory and updated as necessary. During machine learning, the parameter values ​​are adjusted (optimized) in a direction that minimizes the error (loss) calculated based on the learning data.

[0052] The functions of the components constituting the sequence data processing device 1 are as follows.

[0053] The control unit 1110 controls the overall operation of the sequence data processing device 1. Specifically, the control unit 1110 performs control by distinguishing between whether the sequence data processing device 1 operates in a learning mode for adjusting (optimizing) internal parameters, or in an estimation mode for actually estimating coefficients and sequence value data using learned internal parameters. When operating in the learning mode, the control unit 1110 supplies prepared learning data, and performs control to adjust (optimize) the values ​​of the internal parameters held by the coefficient estimation unit 1111 and the internal parameters held by the sequence value estimation unit 1112, based on the overall error calculated by the overall error calculation unit 1116 based on the learning data. In the learning mode, for example, an error backpropagation method is used. When operating in the estimation mode, the control unit 1110 performs control by adjusting (optimizing) the previous context (X <k ) so that the sequence data processing device 1 estimates unknown sequence values ​​based on the estimated sequence values. When operating in the estimation mode, the control unit 1110 provides the learned internal parameter values ​​to the coefficient estimation unit 1111 and the sequence value estimation unit 1112.

[0054] In other words, the control unit 1110 controls the learning of the sequence data processing device 1, so that the sequence data processing device 1 can reduce the coefficient error (LC k ) and the sequence value error (LV k) based on which the values ​​of the learnable internal parameters of the model provided in the coefficient estimation unit 1111 and the values ​​of the learnable internal parameters of the model provided in the sequence value estimation unit 1112 are adjustable.

[0055] The coefficient estimation unit 1111 estimates the previous context (X <k ) based on the coefficient C´ at time k k The coefficient estimation unit 1111 performs processing of a function f(), which will be described later. That is, the coefficient estimation unit 1111 estimates the first sequence data (X k ; 1 ≦ k ≦ L) <k ), the value (X ≧k ) is the estimated coefficient of the approximation function (C´ k ) is provided. The coefficient estimation unit 1111 is configured so that the internal parameters can be learned. That is, the coefficient estimation unit 1111 is realized using, for example, a neural network.

[0056] The sequence value estimation unit 1112 estimates the coefficient C′ at time k estimated by the coefficient estimation unit 1111. k Using this, the series value X after time k is ≧k The estimate of X´ k ,X´ k+1 ,X´ k+2 , . . . The sequence value estimation unit 1112 performs processing of a function g(), which will be described later. During learning, the sequence value estimation unit 1112 calculates the coefficients (C k ) to find the value (X ≧k ) is calculated. The sequence value estimator 1112 is provided with a model for this purpose. The sequence value estimator 1112 is configured so that the internal parameters are trainable. In other words, the sequence value estimator 1112 is realized using, for example, a neural network.

[0057] When the sequence data processing device 1 operates in the learning mode, the coefficient calculation unit 1113 calculates a known sequence value (correct answer) X≧k Based on the coefficient C k The coefficient calculation unit 1113 calculates the coefficient C k That is, the coefficient calculation unit 1113 calculates the value (X ≧k ) based on the correct answer, the value (X ≧k ) is the coefficient of the approximation function (C k ) is calculated. The coefficient calculation unit 1113 uses the above-mentioned HiPPO method. Note that the HiPPO method itself is an existing technology. The coefficient calculation unit 1113 may be called a "HiPPO RNN processing unit."

[0058] When the sequence data processing device 1 operates in the learning mode, the coefficient error calculation unit 1114 calculates the coefficient C estimated by the coefficient estimation unit 1111. k The estimated value of C´ k and the coefficient C calculated by the coefficient calculation unit 1113. k The coefficient error (LC k The coefficient error calculated by the coefficient error calculation unit 1114 is used to adjust (optimize) the parameters of the sequence data processing device 1.

[0059] The sequence value error calculation unit 1115 calculates the estimated value X′ k ,X´ k+1 , and the correct answer for the sequence data (first sequence data) is X k ,X k+1 , . . . In other words, the sequence value error calculation unit 1115 calculates the value (X ≧k ) and the value after position k of the first sequence data (X ≧k ) and the series value error (LV k The sequence value error calculated by the sequence value error calculation unit 1115 is used to adjust (optimize) the parameters of the sequence data processing device 1.

[0060] The overall error calculation unit 1116 calculates an overall error based on the coefficient error calculated by the coefficient error calculation unit 1114 and the sequence value error calculated by the sequence value error calculation unit 1115. As an example, the overall error calculation unit 1116 may calculate the sum of the coefficient error and the sequence value error as the overall error. Using the overall error calculated by the overall error calculation unit 1116, the parameters of the sequence data processing device 1 are adjusted (optimized).

[0061] [Description of the method according to the first embodiment] The sequence data processing device 1 of this embodiment is a special autoregressive model device that inputs a sequence (before time t) that is a prior context, and predicts a subsequent sequence (after time t) that is, in principle, infinitely long. The processing by the sequence data processing device 1 can be formulated as shown in the following equations (8) and (9).

[0062]

number

[0063]

number

[0064] X in formula (8) and formula (9) k is the output value (vector) at time k. Also, X <k is a sequence of previous context values ​​(vectors) before time k, and in principle has an infinite length. In other words, X <k is, ,X k-3 ,X k-2 ,X k-1 It is. However, whether or not the entire sequence of infinitely long previous context values ​​is taken into account when the sequence data processing device 1 predicts the subsequent sequence depends on the configuration method of the base autoregressive model. f() is a (nonlinear) function realized by some (deep) neural network, and usually includes a learnable parameter. Specific examples of the function f include an RNN such as LSTM, a Transformer, etc. k is X at time k <k A sequence X of infinite length that can be derived from k ,X k+1 ,X k+2 ,...,represents the coefficients of an orthonormal basis that approximates the continuous function according to some measure function. k Then, using some function g(), output value (vector) X at time k is calculated. k It is possible to take out E t is the error term. In a typical autoregressive model, C k Directly without going through X k In the method of this embodiment, the entire following context, which is infinite in principle, is estimated once, and then C is calculated as a definite prediction at time k. k From X k This is what distinguishes this model from a typical autoregressive model and is its distinctive feature.

[0065] On the other hand, C k can be calculated by using the RNN embodied in the HiPPO method described above, in principle by storing a sequence value (vector) in the reverse direction of time progression from the infinite future, as shown in the following equation (10).

[0066]

number

[0067] however,

number

[0068] The functions of the various components in the configuration of the sequence data processing device 1 shown in FIG. 2 are as follows, and the above-mentioned method is realized by this functional configuration.

[0069] The coefficient estimation unit 1111 is a neural network (deep neural network) capable of configuring some kind of autoregressive model. The coefficient estimation unit 1111 can be realized by using an RNN such as LSTM, a Transformer, etc. The coefficient estimation unit 1111 realizes a function f() shown in the following equation (12).

[0070]

number

[0071] In other words, the function f() realized by the coefficient estimation unit 1111 is a sequence of prior context values ​​(vectors) X <k Based on this, we obtain the estimated coefficient (predicted value), C´ k Here, the number of input dimensions is D, the orthonormal basis order of HiPPO is N, and the formulas are as shown in (13) and (14) below.

[0072]

number

[0073]

number

[0074] The coefficient calculation unit 1113 calculates the input X k Based on this, the coefficient C kIt is realized as a HiPPO RNN that calculates. Note that A(bar) and B(bar) are calculated in advance after Δ is given as a hyper parameter. Note that, based on the above D and N, it is as shown in the following equations (15) and (16).

[0075]

number

[0076]

number

[0077] The sequence value estimation unit 1112 is a function realized by some kind of neural network, and has a coefficient C k Using this, the function g() expressed by the following equation (17) is calculated.

[0078]

number

[0079] That is, the sequence value estimation unit 1112 calculates the coefficient C k Based on X k The estimate of X´ k When calculating the function g(), the sequence value estimation unit 1112 calculates the coefficient C k The estimated value of C´ k X' may also be used. k is as shown in equation (18) below.

[0080]

number

[0081] The coefficient error calculation unit 1114 calculates an error that contributes to the adjustment (optimization) of the internal parameters of the sequence data processing device 1. That is, the coefficient error calculation unit 1114 calculates C′ calculated by the coefficient estimation unit 1111. k and the coefficient C calculated by the coefficient calculation unit 1113.k The error (for example, the mean square error) between

[0082] The sequence value error calculation unit 1115 calculates an error that contributes to the adjustment (optimization) of the internal parameters of the sequence data processing device 1. That is, the sequence value error calculation unit 1115 calculates an error that contributes to the adjustment (optimization) of the internal parameters of the sequence data processing device 1. k And the correct answer is X k The method of calculating the error by the sequence value error calculation unit 1115 may depend on the task.

[0083] [Learning Procedure of the First Embodiment] Next, the learning procedure of the sequence data processing device 1 will be described. In practice, the length of the sequence to be learned is finite, so a finite-length sequence X 1 ,X 2 ,···,X L The procedure for learning using the above as learning data will be described below.

[0084] Fig. 3 is a flowchart showing the learning procedure of the sequence data processing device 1. The sequence data processing device 1 performs, for example, batch learning or mini-batch learning. The sequence data processing device 1 performs the process shown in the flowchart of Fig. 3 for each piece of sequence data included in a batch or mini-batch. The procedure will be described below with reference to this flowchart.

[0085] Step S1: The coefficient calculation unit 1113 calculates the learning data sequence X 1 ,X 2 ,···,X L By inputting in reverse order, the coefficient C L ,C L-1 ,···,C 1 are calculated in sequence.

[0086] Step S2: For each i for k=1, 2, . . . , L, the following steps S2-1 to S2-5 are performed.

[0087] Step S2-1: The coefficient estimation unit 1111 calculates X<k Based on this, the coefficient estimate C´ k That is, it is calculated as shown in the following formula (19).

[0088]

number

[0089] Step S2-2: The coefficient error calculation unit 1114 calculates the coefficient C k and the coefficient estimate C' calculated in step S2-1. k LC k Calculate.

[0090] Step S2-3: The sequence value estimation unit 1112 calculates the coefficient C k Based on this, the series estimate X' k That is, it is calculated as shown in the following formula (20).

[0091]

number

[0092] Step S2-4: The sequence value error calculation unit 1115 calculates the value X k and the predicted value X' calculated in step S2-3. k LV, which is the error between k Calculate.

[0093] Step S2-5: The total error calculation unit 1116 calculates the total error L k Calculate the total error L k are the errors LC calculated in steps S2-2 and S2-4, respectively. k and L.V. k It is calculated by the following equation (21).

[0094]

number

[0095] Step S3: L, the sum of the total errors for all k 1 +L 2 +···+L L The parameters (learnable parameters) inside the coefficient estimation unit 1111 (function f) and the sequence value estimation unit 1112 (function g) are adjusted so as to minimize the value of . As a method for adjusting the parameters, the backpropagation method can be used.

[0096] [Operation of sequence data processing device 1 after learning is completed] After the above learning is completed using the learning data, the values ​​of the internal parameters of the sequence data processing device 1 are optimized according to the learning data. In this state, the sequence data processing device 1 operates as follows. That is, the coefficient estimation unit 1111 is provided with a prior context X <k The coefficient estimation unit 1111 uses the optimized internal parameters to estimate the prior context X <k Based on this, the coefficient C´ k The coefficient estimation unit 1111 estimates the coefficient C' k The sequence value estimation unit 1112 passes the coefficient C' passed from the coefficient calculation unit 1113 to the sequence value estimation unit 1112. k Using the optimized internal parameters, we calculate the predicted value of the series data, X' k The sequence value estimation unit 1112 calculates the coefficient C' k Using the optimized internal parameters, the predicted value X' after the k+1th k+1 ,X´ k+2 ,··· can also be calculated.

[0097] [Second embodiment] Next, a second embodiment of the present invention will be described. Note that the matters already described in the previous embodiment may not be described below. Here, the matters unique to this embodiment will be mainly described.

[0098] In the second embodiment, the input to the coefficient calculation unit 1113 is a given sequence value X kInstead, it is some output value from a neural network that constitutes an autoregressive model. Specifically, it is as follows.

[0099] [Configuration of sequence data processing device according to the second embodiment] 4 is a block diagram showing a schematic functional configuration of the sequence data processing device according to this embodiment. As shown in the figure, the sequence data processing device 2 includes a control unit 2110, a coefficient estimating unit 2111, a sequence value estimating unit 2112, a coefficient calculating unit 2113, a coefficient error calculating unit 2114, a sequence value error calculating unit 2115, and a total error calculating unit 2116. Each function of the sequence data processing device 2 can also be realized by a computer and a program, or a dedicated electronic circuit.

[0100] The second embodiment is characterized in that the coefficient calculation unit 2113 calculates the sequence data y k (2nd series data) based on the coefficient (C k )

[0101] The control unit 2110 controls the overall operation of the sequence data processing device 2. The control unit 2110 performs control by distinguishing whether the sequence data processing device 2 operates in a learning mode for adjusting (optimizing) internal parameters or in a prediction mode for actually predicting sequence data using learned internal parameters. In particular, when the sequence data processing device 2 is learning, the control unit 2110 performs control by dividing the coefficient error (LC k ) and the sequence value error (LV k ) based on this, control is performed to make adjustable the values ​​of the learnable internal parameters of the model provided in the coefficient estimation unit 2111 and the values ​​of the learnable internal parameters of the model provided in the sequence value estimation unit 2112.

[0102] The coefficient estimation unit 2111 estimates the first sequence data (X k ; 1 ≦ k ≦ L) <k ), the value (X ≧k) is the estimated coefficient of the approximation function (C´ k ) and furthermore, a predetermined value (y k-1 That is, the coefficient estimation unit 2111 performs processing of the function f() in this embodiment.

[0103] The sequence value estimation unit 2112 estimates sequence values ​​based on the coefficients estimated by the coefficient estimation unit 2111. During learning, the sequence value estimation unit 2112 estimates sequence values ​​based on the coefficients (C k ) to calculate the value (X ≧k ) is provided. Note that the internal parameters of the model can be learned. In other words, the sequence value estimator 2112 performs processing of the function g() in this embodiment.

[0104] The coefficient calculation unit 2113 calculates the predetermined value (y k-1 Based on the second sequence data, which is a sequence of y ≧k ), the value (X ≧k ) is the coefficient of the approximation function (C k The coefficient calculation unit 2113 is also called a "HiPPO RNN processing unit."

[0105] The coefficient error calculation unit 2114 calculates the estimated value (C′ k ) and the coefficient (C k ) and the coefficient error (LC k ) is calculated.

[0106] The sequence value error calculation unit 2115 calculates the value (X ≧k ) and the value after position k of the sequence data (X ≧k ) and the series value error (LV k ) is calculated.

[0107] The total error calculation unit 2116 calculates a total error based on the coefficient error and the sequence value error, as in the first embodiment. The total error may be, for example, the sum of the coefficient error and the sequence value error.

[0108] [Description of the Method of the Second Embodiment] The method using the configuration shown in FIG. 4 will be described, focusing on the differences from the first embodiment.

[0109] The coefficient estimation unit 2111 is a neural network (deep neural network) capable of configuring some kind of autoregressive model. The coefficient estimation unit 2111 can be realized by using an RNN such as LSTM, a Transformer, etc. The coefficient estimation unit 2111 realizes a function f shown in the following equation (22).

[0110]

number

[0111] In other words, the previous context X <k The output value of the function f() calculated based on k and y k-1 The vector y output from the function f() k-1 The number of dimensions D' of y can be different from the number of dimensions D of the input values. k-1 is as shown in equation (23) below.

[0112]

number

[0113] The coefficient calculation unit 2113 (HiPPO RNN processing unit) calculates y k-1 Based on the coefficient C k Calculate the coefficient C k is as shown in equation (24) below.

[0114]

number

[0115] [Learning procedure of the second embodiment] Next, a learning procedure of the sequence data processing device 2 will be described. As in the first embodiment, the sequence data processing device 2 is trained using a finite-length sequence. However, in the second embodiment, a finite-length sequence X 1 ,X 2 ,···,X L ,X L+1 It is.

[0116] Fig. 5 is a flowchart showing the learning procedure of the sequence data processing device 2. In this embodiment, too, the sequence data processing device 2 performs the process shown in the flowchart of Fig. 5 on each piece of sequence data included in a batch or a mini-batch. The procedure will be described below with reference to this flowchart.

[0117] Step S11: For each i=1, 2, . . . , L, L+1, the coefficient estimation unit 2111 calculates X <k Based on this, C´ k and y k-1 In other words, in this step, y 0 ,y 1 ,···,y L-1 ,y L is obtained.

[0118] Step S12: y calculated in step S11 0 ,y 1 ,···,y L-1 ,y L By inputting in reverse order, the coefficient calculation unit 2113 (HiPPO RNN processing unit) calculates the coefficient C L ,C L-1 ,···,C 1 are calculated in sequence.

[0119] Step S13: The sequence value estimation unit 2112 calculates the coefficient C 1,C 2 ,···,C L For each k = 1, 2, . . . , L, we use the following formula to find the estimated (predicted) value of the sequence data: X´ k Calculate.

[0120] Step S14: Calculate the error for each k, k=1, 2, . . . , L. That is, the coefficient error calculation unit 2114 calculates C' k and the coefficient C calculated in step S12 k Based on this, for each k = 1, 2, . . . , L, the coefficient error L k In addition, the sequence value error calculation unit 2115 calculates the estimated value X' of the sequence data calculated in step S13. k And the correct answer is given as sequence data X k Based on this, for each k = 1, 2, . . . , L, the sequence value error LV k The total error calculation unit 1116 calculates the coefficient error LC k and the series value error LV calculated by the series value error calculation unit 2115. k By adding and, the total error L k Calculate.

[0121] Step S15: The total sum L of the errors calculated in step S14 1 +L 2 +···+L L The internal parameters of the coefficient estimator 2111 (function f) and the sequence value estimator 2112 (function g) are adjusted (optimized) so as to minimize

[0122] [Operation of sequence data processing device 2 after learning is completed] After the above learning is completed using the learning data, the values ​​of the internal parameters of the sequence data processing device 2 are optimized according to the learning data. In this state, the sequence data processing device 2 operates as follows. That is, the coefficient estimation unit 2111 is provided with the previous context X <kThe coefficient estimation unit 2111 uses the optimized internal parameters to estimate the prior context X <k Based on this, the coefficient C´ k The coefficient estimation unit 2111 estimates the coefficient C' k The sequence value estimation unit 2112 receives the coefficient C k Using the optimized internal parameters, we calculate the predicted value of the series data, X' k The sequence value estimation unit 2112 calculates the coefficient C' k Using the optimized internal parameters, the predicted value X' after the k+1th k+1 ,X´ k+2 ,··· can also be calculated.

[0123] [Third embodiment] Next, a third embodiment of the present invention will be described. Note that the description of the matters already described in the previous embodiments may be omitted below. Here, the description will focus on matters unique to this embodiment.

[0124] [Configuration of sequence data processing device according to the third embodiment] 6 is a block diagram showing a schematic functional configuration of the sequence data processing device according to this embodiment. As shown in the figure, the sequence data processing device 3 includes a control unit 3110, a distribution statistical information estimation unit 3161, a distribution statistical information calculation unit 3162, a sampling processing unit 3163, a coefficient estimation unit 3164, a KL divergence error calculation unit 3165, a sequence value estimation unit 3112, a coefficient calculation unit 3113, a coefficient error calculation unit 3114, a sequence value error calculation unit 3115, and a total error calculation unit 3116. The functions of the sequence data processing device 3 may be realized by a computer and a program, or by a dedicated electronic circuit, as in the first and second embodiments.

[0125] The control unit 3110 controls the overall operation of the sequence data processing unit 3. The control unit 3110 performs control by distinguishing whether the sequence data processing unit 3 operates in a learning mode for adjusting (optimizing) internal parameters or in a prediction mode for actually predicting sequence data using learned internal parameters. When operating in the learning mode, the control unit 3110 performs control to adjust the values ​​of the internal parameters held by the distribution statistical information calculation unit 3162, the internal parameters held by the distribution statistical information estimation unit 3161, the internal parameters held by the coefficient estimation unit 3164, and the internal parameters held by the sequence value estimation unit 3112, based on the overall error calculated by the overall error calculation unit 3116. By the control unit 3110 performing the control, the sequence data processing device 3 is configured to be able to adjust the learnable internal parameter values ​​of the model provided in the distribution statistical information calculation unit 3162, the learnable internal parameter values ​​of the model provided in the distribution statistical information estimation unit 3161, the learnable internal parameter values ​​of the model provided in the coefficient estimation unit 3164, and the learnable internal parameter values ​​of the model provided in the sequence value estimation unit 3112 based on the coefficient error, the sequence value error, and the distribution statistical information error.

[0126] The distribution statistical information estimation unit 3161 estimates a previous context (X <k ) is used to calculate the coefficient of the approximation function that approximates the series data (C k ; 1≦k≦L). Specifically, the distribution statistical information estimation unit 3161 calculates a distribution statistical information estimate that is an estimate of distribution statistical information that is statistical information regarding the distribution of the first sequence data at position k. <k ), the value (X ≧k ) is the coefficient of the approximation function (C kThe distribution statistical information estimation unit 3161 is configured to include a model for calculating a distribution statistical information estimate, which is an estimate of distribution statistical information, which is statistical information regarding the distribution of (a,b,c,c,d,e,e,i). An example of distribution statistical information is the mean value and standard deviation in a Gaussian distribution. The distribution statistical information estimation unit 3161 performs processing of a function f(), which will be described later. The distribution statistical information estimation unit 3161 is configured to be able to learn internal parameters. That is, the distribution statistical information estimation unit 3161 can be realized by using, for example, a neural network.

[0127] The distribution statistical information calculation unit 3162 calculates the k-th coefficient (C k ;1≦k≦L) based on the coefficient (C k In other words, the distribution statistical information calculation unit 3162 calculates the coefficient (C k ;1≦k≦L) based on the coefficient (C k The distribution statistical information calculation unit 3162 is configured with a model for calculating distribution statistical information, which is statistical information regarding the distribution of k (l, k=1, 1≦k≦L). An example of the distribution statistical information is the mean value and standard deviation in a Gaussian distribution. The distribution statistical information calculation unit 3162 processes a function e(), which will be described later. The distribution statistical information calculation unit 3162 is configured so that the internal parameters of the model can be learned. In other words, the distribution statistical information calculation unit 3162 can be realized by using, for example, a neural network.

[0128] The sampling processing unit 3163 generates samples in the distribution represented by the given distribution statistical information. When generating samples, the sampling processing unit 3163 appropriately uses a random number generator or a pseudo-random number generator.

[0129] The coefficient estimation unit 3164 derives a k-th coefficient (C k ) estimate (C´ kThat is, the coefficient estimation unit 3164 estimates the samples (Z k ), for a given k (1≦k≦L) of the first sequence data, the kth coefficient of the approximation function (C k ) estimate (C´ k The coefficient estimation unit 3164 is configured to include a model for estimating a function d(), which will be described later. The coefficient estimation unit 3164 is configured to be able to learn the internal parameters of the model. That is, the coefficient estimation unit 3164 can be realized by using, for example, a neural network.

[0130] The KL divergence error calculation unit 3165 calculates a KL divergence error (distribution statistical information error, LK k ) is calculated. The KL divergence error calculation unit 3165 is also called a "distribution statistical information error calculation unit." For example, the KL divergence error calculation unit 3165 calculates the KL divergence error based on the mean and variance of both distributions.

[0131] The sequence value estimation unit 3112 calculates an estimate (X' k During learning, the sequence value estimation unit 3112 estimates the k-th coefficient (C k ) to find the kth value of the sequence data (X k The sequence value estimation unit 3112 performs processing to estimate function g(), which will be described later. The sequence value estimation unit 3112 is configured so that the internal parameters of the model are trainable. In other words, the sequence value estimation unit 3112 is realized using, for example, a neural network.

[0132] The coefficient calculation unit 3113 calculates the first sequence data (Xk where k is an integer representing the order of the sequence, and 1≦k≦L is used to calculate the coefficient (C k ; 1≦k≦L. The coefficient calculation unit 3113 is also called a "HiPPO RNN processing unit."

[0133] The coefficient error calculation unit 3114 calculates the k-th coefficient (C k ) estimate (C´ k ) and the k-th coefficient (C k ) and the coefficient error (LC k ) is calculated.

[0134] The sequence value error calculation unit 3115 calculates an estimate (X' k ) and the k-th value of the sequence data given as training data (X k ) and calculate the estimate error.

[0135] The total error calculation unit 3116 calculates the coefficient error (LC k ) and the series value error (LV k ) and the distribution statistical information error (KL divergence error LK k ) is calculated. Based on this overall error, parameters in the sequence data processor 3 are adjusted (optimized). An example of the overall error is the sum of the coefficient error, sequence value error, and distribution statistical information error.

[0136] [Description of the method according to the third embodiment] The method using the configuration shown in FIG. 6 will be described with a focus on the differences from the first embodiment. In the first embodiment, the coefficients of the autoregressive model (C k), whereas the sequence data processor 3 of the third embodiment estimates the distribution (probability distribution) of coefficients. The method according to this embodiment when using a VAE (Variational Autoencoder) is as follows.

[0137] The distribution statistical information estimation unit 3161 is configured using some kind of neural network (deep neural network). The distribution statistical information estimation unit 3161 calculates a function f(). The function f() is expressed by the following formula (25).

[0138]

number

[0139] μ´ in Eq. (25) k and σ′ k are the latent variables Z' k These are estimates (predictions) of the mean and standard deviation in the distribution (Gaussian distribution) of k and σ′ k is as shown in equation (26) below.

[0140]

number

[0141] where D´´ is the input value X t In general, for example, D´ <Dであってよい。

[0142] The configuration of the sequence data processing device 3 shown in FIG. 6 has the following features. In the first embodiment, the coefficient estimation unit 1111 calculates X <k Based on the coefficient C k On the other hand, the features of the functions of the sequence data processor 3 are as follows.

[0143] In this embodiment, X <kBased on this, the distribution statistical information estimation unit 3161 estimates the latent variable Z′ k The mean value μ´ of the distribution (Gaussian distribution) k and standard deviation σ´ k That is, the distribution statistical information estimation unit 3161 is configured as some kind of neural network (deep neural network) that realizes the function f() expressed by the above equation (25).

[0144] On the other hand, the distribution statistical information calculation unit 3162 calculates the coefficient C k Based on this, the latent variable Z k The average value μ k and standard deviation σ k The distribution statistical information calculation unit 3162 is configured using some kind of neural network (deep neural network) and calculates the coefficient C k From the average value μ k and standard deviation σ k The internal parameters of the neural network (deep neural network) of the distribution statistical information calculation unit 3162 are learnable parameters. The processing of the projection function e() by the distribution statistical information calculation unit 3162 is expressed by the following formula (27). Note that the average value μ k and standard deviation σ k is as shown in equation (28) below.

[0145]

number

[0146]

number

[0147] The sampling processing unit 3163 also calculates the average value μ k and standard deviation σ kIn other words, the sampling processing unit 3163 appropriately uses a random number generator or a pseudo-random number generator to perform sampling processing in the distribution. k and standard deviation σ k (Both are D´´-dimensional vectors) k Generate a sample of

[0148] In addition, the coefficient estimation unit 3164 estimates the latent variable Z k Based on the sample, the coefficient C k The estimated value of C´ k The coefficient estimation unit 3164 is configured as some kind of neural network (deep neural network) and calculates Z k From C´ k The coefficient estimation unit 3164 realizes a projection function d() onto the neural network. Note that the internal parameters of the neural network (deep neural network) included in the coefficient estimation unit 3164 are learnable parameters. The processing of the projection function d() by the coefficient estimation unit 3164 is expressed by the following equation (29).

[0149]

number

[0150] Then, the coefficient error calculation unit 3114 calculates the coefficient C output by the coefficient calculation unit 3113 (HiPPO RNN processing unit). k and the coefficient estimate C' output by the coefficient estimation unit 3164. k Calculate the error (coefficient error) between

[0151] In this embodiment, the neural network is trained based on the KL divergence error as well. The KL divergence error represents the difference between distributions. The KL divergence error is also called the "distribution statistical information error." That is, the KL divergence error calculation unit 3165 calculates the μ k and σ kand μ′, which is an estimated value output by the distribution statistical information estimation unit 3161. k and σ′ k Calculate the error (KL divergence error) between

[0152] As in the first embodiment, the sequence value estimation unit 3112 calculates the coefficient C k Based on this, the estimated value of the series data X' k Then, the sequence value error calculation unit 3115 calculates the input sequence value X k and X' output by the sequence value estimation unit 3112. k Calculate the error (series value error) between

[0153] The overall error calculation unit 3116 calculates an overall error which is the result of combining the coefficient error calculated by the coefficient error calculation unit 3114, the sequence value error calculated by the sequence value error calculation unit 3115, and the KL divergence error calculated by the KL divergence error calculation unit 3165.

[0154] During learning, the sequence data processing device 3 optimizes the neural network of each part based on the above-mentioned overall error. That is, the sequence data processing device 3 adjusts (optimizes) the values ​​of the internal parameters of the neural network included in each part using the backpropagation method based on the above-mentioned overall error.

[0155] [Learning procedure of the third embodiment] Next, the learning procedure of the sequence data processor 3 will be described. Fig. 7 is a flowchart showing the learning procedure of the sequence data processor 3. The sequence data processor 3 performs the process shown in the flowchart of Fig. 7 on each piece of sequence data included in a batch or a mini-batch. The procedure will be described below with reference to this flowchart.

[0156] Step S21: The coefficient calculation unit 3113 (HiPPO RNN processing unit) calculates the sequence X 1 ,X 2 ,···,X LBy inputting in reverse order, the coefficient C L ,C L-1 ,···,C 1 The calculated coefficient C L ,C L-1 ,···,C 1 The value of is passed to the distribution statistical information calculation unit 3162, the sequence value estimation unit 3112, and the coefficient error calculation unit 3114 for subsequent processing.

[0157] Step S22: For each i=1, 2, . . . , L, the following steps S22-1 to S22-9 are performed.

[0158] Step S22-1: The distribution statistical information calculation unit 3162 processes the function e() to obtain the input C k Based on μ k and σ k Calculate.

[0159] Step S22-2: The sampling processing unit 3163 calculates the latent variable Z k That is, the sampling processing unit 3163 samples an instance of the latent variable Z k Here, ε is a normal distribution (Gaussian distribution) random value generated independently of all other dimensions. The sampling processing unit 3163 may use any random number generator or pseudorandom number generator realized by existing technology to generate the random value ε. Here, ε is as shown in the following formula (31).

[0160]

number

[0161]

number

[0162] Step S22-3: The coefficient estimation unit 3164 calculates the value Z k The projection function d() is calculated based on k That is, the coefficient estimation unit 3164 performs the process of the following equation (32).

[0163]

number

[0164] Step S22-4: The coefficient error calculation unit 3114 calculates the coefficient C k and the coefficient estimate C', which is the result of estimation by the coefficient estimation unit 3164 in step S22-3. k The error calculated in this step by the coefficient error calculation unit 3114 is the coefficient error LC k It is.

[0165] Step S22-5: The sequence value estimation unit 3112 calculates the coefficient C k Based on this, the function g() is calculated to obtain the estimated value X' of the sequence data. k That is, the sequence value estimator 3112 performs the process of the following equation (33).

[0166]

number

[0167] Step S22-6: The sequence value error calculation unit 3115 calculates the value X k and the predicted value X′ of the sequence calculated by the sequence value estimator 3112 in step S22-5. k The error calculated in this step by the sequence value error calculation unit 3115 is the sequence value error LV k It is.

[0168] Step S22-7: The distribution statistical information estimation unit 3161 calculates the distribution statistical information X <k Based on this, the mean value of the distribution is μ´ k and standard deviation σ´ k That is, the distribution statistical information estimation unit 3161 estimates the previous context X <k The calculation process of the distribution statistical information estimation unit 3161 in this step is as shown in the following equation (34).

[0169]

number

[0170] Step S22-8: The KL divergence error calculation unit 3165 calculates μ k and σ k and μ′ calculated in step S22-7 k and σ′ k The error calculated in this step by the KL divergence error calculation unit 3165 is the KL divergence error LK k and can be calculated by the following equation (35):

[0171]

number

[0172] In addition, "KL()" on the right side of equation (35) is a function that returns the Kullback-Leibler divergence. In equation (35), the function KL() is the mean value μ k with a standard deviation of σ k A normal distribution with mean μ´ k and the standard deviation is σ´ k Returns a number that represents the difference between a normal distribution and

[0173] Step S22-9: The total error calculation unit 3116 calculates the coefficient error (LC k ) and the series value error (LV k ) and the KL divergence error (LK k ) and the total error L k The total error calculation unit 3116 calculates the total error L for the current value of k by, for example, the following equation (36): k Calculate.

[0174]

number

[0175] Total error L k The value of does not necessarily have to be calculated by equation (36). For example, in equation (36), the coefficient error (LC k ) and series value error (LV k ) and the KL divergence error (LK k ) and add them together to obtain the total error L k may be calculated.

[0176] Step S23: For all k = 1, 2, . . . , L, the total error L k After the L is obtained, the control unit 3110 k The learnable parameters of functions f(), g(), d(), and e() are tuned (optimized) and updated to minimize the sum of

[0177] [Operation of sequence data processing device 3 after learning completion] After the above learning is completed using the learning data, the values ​​of the internal parameters of the sequence data processing device 3 are optimized according to the learning data. In this state, the sequence data processing device 3 operates as follows. That is, the distribution statistical information estimation unit 3161 is provided with the previous context X <kThe distribution statistical information estimation unit 3161 uses the optimized internal parameters to estimate the previous context X <k Based on this, distribution statistics (average value μ k and standard deviation σ k The distribution statistical information estimation unit 3161 estimates the average value μ′ k and standard deviation σ´ k The sampling processing unit 3163 uses the estimated value of the distribution statistical information passed from the distribution statistical information estimation unit 3161 and also uses a random number generator or the like as appropriate to generate a sample value Z k The sampling processing unit 3163 generates the generated sample value Z k The coefficient estimation unit 3164 receives the sample value Z k Based on this and using the optimized internal parameters, we estimate the coefficients of the approximation function of the series data, C´ k The coefficient estimation unit 3164 estimates the obtained coefficient estimate C' k The sequence value estimation unit 3112 passes the coefficient estimate C' passed from the coefficient estimation unit 3164 to the sequence value estimation unit 3112. k Using the optimized internal parameters, we calculate the predicted value of the series data, X' k The sequence value estimation unit 3112 calculates the coefficient C' k Using the optimized internal parameters, the predicted value X' after the k+1th k+1 ,X´ k+2 ,··· can also be calculated.

[0178] Next, a number of modified examples of the embodiment will be described. Note that, as long as they can be combined, a number of modified examples may be combined and implemented.

[0179] [First Modification] The first modified example is a modified example of the third embodiment. Specifically, the first modified example combines the configuration of the third embodiment with the features of the second embodiment. This first modified example will be described next.

[0180] 8 is a block diagram showing a schematic functional configuration of a sequence data processing device according to a first modified example. As shown in the figure, the sequence data processing device 4 includes a control unit 4110, a distribution statistical information estimation unit 4161, a distribution statistical information calculation unit 4162, a sampling processing unit 4163, a coefficient estimation unit 4164, a KL divergence error calculation unit 4165, a sequence value estimation unit 4112, a coefficient calculation unit 4113, a coefficient error calculation unit 4114, a sequence value error calculation unit 4115, and a total error calculation unit 4116. The functions of the sequence data processing device 4 may be realized by a computer and a program, or by a dedicated electronic circuit, as in the first, second, and third embodiments.

[0181] The functional configuration of the sequence data processing device 4 according to this modification is similar to that of the sequence data processing device 3 described in the third embodiment. That is, the control unit 4110 has the same or similar function as the control unit 3110 in the third embodiment. Moreover, the distribution statistical information estimation unit 4161 has the same or similar function as the distribution statistical information estimation unit 3161 in the third embodiment. Moreover, the distribution statistical information calculation unit 4162 has the same or similar function as the distribution statistical information calculation unit 3162 in the third embodiment. Moreover, the sampling processing unit 4163 has the same or similar function as the sampling processing unit 3163 in the third embodiment. Moreover, the coefficient estimation unit 4164 has the same or similar function as the coefficient estimation unit 3164 in the third embodiment. Moreover, the KL divergence error calculation unit 4165 (distribution statistical information error calculation unit) has the same or similar function as the KL divergence error calculation unit 3165 in the third embodiment. Moreover, the sequence value estimation unit 4112 has a function that is the same as or similar to the sequence value estimation unit 3112 in the third embodiment. Moreover, the coefficient calculation unit 4113 (HiPPO RNN processing unit) has a function that is the same as or similar to the coefficient calculation unit 3113 in the third embodiment. Moreover, the coefficient error calculation unit 4114 has a function that is the same as or similar to the coefficient error calculation unit 3114 in the third embodiment. Moreover, the sequence value error calculation unit 4115 has a function that is the same as or similar to the sequence value error calculation unit 3115 in the third embodiment. Moreover, the overall error calculation unit 4116 has a function that is the same as or similar to the overall error calculation unit 3116 in the third embodiment.

[0182] The feature of the sequence data processing device 4 is that the distribution statistical information estimation unit 4161 calculates the value (y k-1 ) and output its value (y k-1 ) based on the sequence, the coefficient calculation unit 4113 (HiPPO RNN processing unit) calculates the coefficients of the approximation function (C k ;1≦k≦L).

[0183] The distribution statistical information estimation unit 4161 has the functions of the distribution statistical information estimation unit 3161 in the third embodiment, and also has the functions of the previous context (X <i ), and then a predetermined value (yk-1 ) is calculated. k-1 ) is input to the coefficient calculation unit 4113 (HiPPO RNN processing unit).

[0184] The coefficient calculation unit 4113 (HiPPO RNN processing unit) calculates the predetermined value (y k-1 ), and the predetermined value (y k-1 ) is used to calculate the coefficients (C k ;1≦k≦L). The coefficient calculation unit 4113 (HiPPO RNN processing unit) calculates the value (y k-1 ) is the sequence y 1 ,y 2 ,···,y L By inputting in reverse order, the coefficient C L ,C L-1 ,···,C 2 ,C 1 That is, the coefficient calculation unit 4113 (HiPPO RNN processing unit) calculates the predetermined value (y k-1 Based on the second sequence data, which is a sequence of y ≧k ), the value (X ≧k ) is the coefficient of the approximation function (C k ) is calculated.

[0185] The sequence value error calculation unit 4115 calculates the predicted value (X' k ) and the k-th value of the second series data (y k ) and the series value error (LV k ) is calculated.

[0186] That is, in this first modified example, the distribution statistical information estimation unit 4161 is the same as the distribution statistical information estimation unit 3161 in the third embodiment. kt and σ′ kIn addition to calculating y k-1 That is, the process performed by the distribution statistical information estimation unit 4161 in this first modified example is the calculation of the function f() expressed by the following equation (37).

[0187]

number

[0188] The above y k-1 In the same manner as the coefficient estimation unit 2111 in the second embodiment, the coefficient calculation unit 4113 (HiPPO RNN processing unit) calculates the coefficient of the approximation function C k (1≦k≦L). The coefficient calculation unit 4113 (HiPPO RNN processing unit) calculates y 0 ,y 1 ,···,y L-1 ,y L By inputting the values ​​in reverse order, the coefficient C L ,C L-1 ,···,C 1 The subsequent processing is similar to that of the sequence data processing device 3 described in the third embodiment.

[0189] [Second modified example] In the third embodiment, the average value and standard deviation in the distribution are used as the distribution statistical information. The average value and variance may also be used as the distribution statistical information. Furthermore, other numerical values ​​or information that represent the distribution pattern may also be used as the distribution statistical information.

[0190] [Third Modification] In each embodiment, the processing procedure has been described with reference to the flowcharts (FIGS. 3, 5, and 7), but the order of these processes is merely an example. The order of the processes may be changed as long as no contradiction occurs in the dependency of the calculations.

[0191] [Fourth Variation] In each embodiment and modification, time series data has been described as an example of sequence data, but the sequence data processed by the sequence data processing device is not limited to time series data. The sequence data processing device of the fourth modification may process any sequence data other than time series data. In the case of time series data, a specific position in the sequence corresponds to a specific time, but in general, a specific position in the sequence may correspond to something other than time.

[0192] [Fifth Modification] In each of the embodiments and modifications, the sequence data processing device has a learning function. In the fifth modification, the sequence data processing device does not have a learning function. That is, in the fifth modification, the sequence data processing device does not adjust parameters of an internal model using learning data. The sequence data processing device of the fifth modification stores learned parameter values, and performs estimation processing on sequence data using these parameter values.

[0193] [Sixth Modification] In each of the embodiments and modifications, the sequence data processing device estimates coefficients based on the preceding context and also estimates sequence values. The sequence data processing device of the sixth modification estimates coefficients based on the preceding context, but does not estimate sequence values. When the sequence data processing device of the sixth modification estimates coefficients of an approximation function of sequence data, the information on the coefficient estimates is equivalent or nearly equivalent to the information on the estimated sequence values.

[0194] FIG. 9 is a block diagram showing an example of an internal configuration for realizing the functions of a sequence data processing device in the first, second, and third embodiments and modifications. The sequence data processing device 1, 2, 3, or 4 can be realized by using a computer shown in FIG. 9. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized by using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. The RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device or the like. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from and to the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port 903 via the bus 906.

[0195] When at least some of the functions of the series data processing device are realized by a computer, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memories, and storage devices such as hard disks built into a computer system. In other words, the term "computer-readable recording medium" may be a non-transitory computer-readable recording medium. The term "computer-readable recording medium" may also include a medium that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. The above-mentioned program may be a program for realizing some of the functions described above, or may be a program that can be realized in combination with a program already recorded in the computer system.

[0196] Although several embodiments and modified examples of the present invention have been described in detail above with reference to the drawings, the specific configurations are not limited to these embodiments, and designs that do not deviate from the gist of the present invention are also included.

[0197] [effect] According to at least one of the above-described embodiments (including modified examples), it is possible to efficiently learn an autoregressive model that performs long-term prediction. In other words, it is possible to efficiently learn a continuous-time autoregressive model, for which problem setting was difficult with conventional techniques, and a high-performance autoregressive model that has the ability to predict beyond the next time.

[0198] [Use and Application] The sequence data processing device described in each of the above embodiments (including the modified examples) can be used to process various sequence data, and is not limited to specific sequence data. Sequence data includes, for example, time series data, data distributed in a spatial direction, linguistic expressions (sequences of characters or words), and numerical data of other dimensions that do not correspond to dimensions such as time or space. Time series data includes, for example, natural phenomena, social phenomena (including but not limited to economic phenomena), and sounds (e.g., spoken language). Data distributed in a spatial direction includes, for example, two-dimensional images, and three-dimensional or higher-dimensional data. Sequence data other than those exemplified here may be processed by the sequence data processing device.

[0199] It should be noted that any literal variables used in the claims are not intended to limit the scope of the claims. [Industrial Applicability]

[0200] The present invention can be used, for example, for processing sequence data in a wide range of applications. There is no specific limitation on the type of sequence data. The scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0201] 1,2,3,4 series data processing device 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus 1110 Control section 1111 Coefficient Estimation Unit 1112 Series value estimation unit 1113 Coefficient calculation unit (HiPPO RNN processing unit) 1114 Coefficient error calculation unit 1115 Series value error calculation unit 1116 Total error calculation unit 2110 Control section 2111 Coefficient Estimation Unit 2112 Series value estimation unit 2113 Coefficient calculation unit (HiPPO RNN processing unit) 2114 Coefficient error calculation unit 2115 Series value error calculation unit 2116 Total error calculation unit 3110 Control section 3112 Series value estimation unit 3113 Coefficient calculation unit (HiPPO RNN processing unit) 3114 Coefficient error calculation unit 3115 Series value error calculation unit 3116 Total Error Calculation Unit 3161 Distribution statistical information estimation section 3162 Distribution statistical information calculation unit 3163 Sampling Processing Unit 3164 Coefficient Estimation Unit 3165 KL Divergence Error Calculation Unit (Distribution Statistical Information Error Calculation Unit) 4110 Control section 4112 Series value estimation unit 4113 Coefficient calculation unit (HiPPO RNN processing unit) 4114 Coefficient error calculation unit 4115 Series value error calculation unit 4116 Total error calculation unit 4161 Distribution statistical information estimation section 4162 Distribution statistical information calculation unit 4163 Sampling Processing Unit 4164 Coefficient Estimation Unit 4165 KL Divergence Error Calculation Unit (Distribution Statistical Information Error Calculation Unit)

Claims

1. The previous context (X ＜ｋ ), the value (X ≧ｋ ) is the estimated coefficient of the approximation function (C' ｋ a coefficient estimator with a model for estimating A sequence data processing device comprising:

2. The value after position k of the first sequence data (X ≧ｋ ) based on the correct answer, the value (X ≧ｋ ) is the coefficient of the approximation function (C ｋ A coefficient calculation unit for calculating The coefficient (C ｋ ) to obtain the value (X ≧ｋ a sequence value estimator including a model for calculating at least a portion of an estimate of The coefficient estimation unit estimates the coefficients (C' ｋ ) and the coefficient (C ｋ ) and the coefficient error (LC ｋ A coefficient error calculation unit that calculates The value (X ≧ｋ ) and the value (X ≧ｋ ) and the series value error (LV ｋ A sequence value error calculation unit that calculates Further equipped with The coefficient error (LC ｋ ), and the sequence value error (LV ｋ ) and based on the above, values ​​of internal parameters that can be learned of the model included in the coefficient estimation unit and values ​​of internal parameters that can be learned of the model included in the sequence value estimation unit are adjustable. The sequence data processing device according to claim 1 .

3. The coefficient estimation unit ＜ｋ ), and further based on a predetermined value (y ｋ－１ ) is calculated, The predetermined value (y ｋ－１ Based on the second sequence data, which is a sequence of y ≧ｋ ), the value (X ≧ｋ ) is the coefficient of the approximation function (C ｋ A coefficient calculation unit for calculating The coefficient (C ｋ ) to obtain the value (X ≧ｋ a sequence value estimator including a model for calculating at least a portion of an estimate of The coefficient estimation unit estimates the coefficients (C' ｋ ) and the coefficient (C ｋ ) and the coefficient error (LC ｋ A coefficient error calculation unit that calculates The value (X ≧ｋ ) and the value (y ≧ｋ ) and the series value error (LV ｋ A sequence value error calculation unit that calculates Further equipped with The coefficient error (LC ｋ ), and the sequence value error (LV ｋ ) and based on the above, values ​​of internal parameters that can be learned of the model included in the coefficient estimation unit and values ​​of internal parameters that can be learned of the model included in the sequence value estimation unit are adjustable. The sequence data processing device according to claim 1 .

4. The previous context (X ＜ｋ ), the value (X ≧ｋ ) is the coefficient of the approximation function (C ｋ a distribution statistical information estimation unit including a model for calculating a distribution statistical information estimate value, which is an estimate value of distribution statistical information, which is statistical information regarding the distribution of A sequence data processing device comprising:

5. The value after position k of the first sequence data (X ≧ｋ ) based on the correct answer, the value (X ≧ｋ ) is the coefficient of the approximation function (C ｋ A coefficient calculation unit for calculating The coefficient (C ｋ ; 1 ≦ k ≦ L) based on the coefficient (C ｋ A distribution statistical information calculation unit including a model for calculating distribution statistical information, which is statistical information regarding the distribution of k (1≦k≦L); The sample (Z ｋ ), for a given k (1≦k≦L) of the first sequence data, the k-th coefficient (C ｋ ) Estimated value (C' ｋ A coefficient estimator having a model for estimating The k-th coefficient (C ｋ ) based on the k-th value of the first sequence data, ｋ A sequence value estimator having a model for estimating The k-th coefficient (C ｋ ) Estimated value (C' ｋ ), and the k-th coefficient (C ｋ ) and the coefficient error (LC ｋ A coefficient error calculation unit that calculates The predicted value (X' ｋ ) and the kth value of the given sequence data (X ｋ ) and the series value error (LV ｋ A sequence value error calculation unit that calculates A distribution statistical information error (LK ｋ A distribution statistical information error calculation unit that calculates Further equipped with a learnable internal parameter value of a model provided in the distribution statistical information calculation unit, a learnable internal parameter value of a model provided in the distribution statistical information estimation unit, a learnable internal parameter value of a model provided in the coefficient estimation unit, and a learnable internal parameter value of a model provided in the sequence value estimation unit are configured to be adjustable based on the coefficient error, the sequence value error, and the distribution statistical information error.

5. The sequence data processing device according to claim 4.

6. The distribution statistical information estimation unit is ＜ｋ ), and further based on a predetermined value (y ｋ－１ ) is calculated, The predetermined value (y ｋ－１ Based on the second sequence data, which is a sequence of y ≧ｋ ), the value (X ≧ｋ ) is the coefficient of the approximation function (C ｋ A coefficient calculation unit for calculating The coefficient (C ｋ ; 1 ≦ k ≦ L) based on the coefficient (C ｋ A distribution statistical information calculation unit including a model for calculating distribution statistical information, which is statistical information regarding the distribution of k (1≦k≦L); The sample (Z ｋ ), for a given k (1≦k≦L) of the first sequence data, the k-th coefficient (C ｋ ) Estimated value (C' ｋ A coefficient estimator having a model for estimating The k-th coefficient (C ｋ ) based on the k-th value of the first sequence data, ｋ A sequence value estimator having a model for estimating The k-th coefficient (C ｋ ) Estimated value (C' ｋ ) and the k-th coefficient (C ｋ ) and the coefficient error (LC ｋ A coefficient error calculation unit that calculates The predicted value (X' ｋ ), and the k-th value of the second sequence data (y ｋ ) and the series value error (LV ｋ A sequence value error calculation unit that calculates A distribution statistical information error (LK ｋ A distribution statistical information error calculation unit that calculates Further equipped with a learnable internal parameter value of a model provided in the distribution statistical information calculation unit, a learnable internal parameter value of a model provided in the distribution statistical information estimation unit, a learnable internal parameter value of a model provided in the coefficient estimation unit, and a learnable internal parameter value of a model provided in the sequence value estimation unit are configured to be adjustable based on the coefficient error, the sequence value error, and the distribution statistical information error.

5. The sequence data processing device according to claim 4.

7. A sequence data processing device according to any one of claims 1 to 6, A program that makes a computer function as a

Images