Extended reactive network device, inference device, and program

Abstract

Provided is a device or the like expanded so that the value of the matrix element Dnm represented by Equation (28) in the remanent network can be flexibly changed in n ≥ m.SOLUTION: The extended reactive network unit includes a network that inputs data X of a matrix representing a sequence of a d-dimensional vector with a length l and estimates data Y of a matrix representing a sequence of a d-dimensional vector with a length l based on the data X. The extended reactive network unit calculates an n-th (1 ≤ n ≤ 1) d-dimensional vector in a series of data Y by applying a coefficient Dnm to an m-th (1 ≤ m ≤ 1) d-dimensional vector in a series of data X. For n> m, 0. ltoreq. D. sub. nm <1, and the larger the value of (n-m), the smaller the value of D. sub. nm, depending on the Iga datum X by a successively decreasing degree from the value of D. sub. nm for m = n-1 to the value of D. sub. nm for m = 1 for a given n.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an extended retentive network device, an inference device, and a program. [Background technology]

[0002] Neural networks are used to infer output data based on input data. Here, "inference" refers to the process of estimating output data corresponding to input data for each of various tasks. Various methods for inference using neural networks have been researched.

[0003] One class of processing performed by neural networks is inference using input data that is a sequence (called sequence inference). In sequence inference, the state of the network and the output from the network are inferred based on the sequence of input data. Examples of network configurations for performing sequence inference are described in Non-Patent Documents 1 to 6.

[0004] Non-Patent Document 1 describes a recurrent neural network (RNN) technology.

[0005] Non-Patent Document 2 describes the technology of an Elman network, which is a type of recurrent neural network and has the most basic configuration.

[0006] Non-Patent Document 3 describes a technology for long short-term memory (LSTM), which is one of the networks with a gate mechanism.

[0007] Non-Patent Document 4 describes a technology of a gated recurrent unit (GRU), which is one of the networks having a gate mechanism.

[0008] Non-Patent Document 5 describes a neural network technology called a Transformer that has an internal attention mechanism.

[0009] Non-Patent Document 6 describes the technology of a Retentive Network, which is a network for solving the problems inherent in self-attention mechanisms. The Retentive Network has a mechanism called retention, which serves as an alternative to the self-attention of a transformer or the like. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams, Learning representations by back-propagating errors, Nature. 323 (6088) , pp.533-536, doi:10.1038 / 323533a0, 1986. [Non-patent document 2] Jeffrey L. Elman, Finding Structure in Time, Cognitive Science, 14 (2) , pp. 179-211, doi:10.1016 / 0364-0213(90)90002-E, 1990. [Non-patent document 3] Sepp Hochreiter, Jurgen Schmidhuber, Long short-term memory, Neural Computation, 9 (8), pp. 1735-1780, doi:10.1162 / neco.1997.9.8.1735, 1997. [Non-patent document 4] Kyunghyun Cho, Bart van Merrienboer, DZmitry Bahdanau, Yoshua Bengio, On the Properties of Neural Machine Translation: Encoder-Decoder Approaches, https: / / arxiv.org / abs / 1409.1259, 2014. [Non-patent document 5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, Attention Is All You Need, In proceeding of the 31st Conference on Neural Information Processing Systems (NIPS 2017), https: / / papers.nips.cc / paper_files / paper / 2017 / hash / 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html, 2017. [Non-patent document 6] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei, Retentive Network: A Successor to Transformer for Large Language Models, In proceeding of the 12th International Conference on Learning Representations (ICLR 2024), https: / / arxiv.org / pdf / 2307.08621, 2023. Summary of the Invention [Problem to be solved by the invention]

[0011] However, the retentive network described in Non-Patent Document 6 has the following problems.

[0012] The configuration of the retentive network will be described in detail later by referring to Equations (23) to (31). In the learning of the retentive network, based on the input data X (a matrix representing a series of length l of d-dimensional vectors), Q, K, and V are respectively calculated. The calculation methods of Q, K, and V are respectively represented by Equations (23), (24), and (25). And the output Y (similarly to the above input X, a matrix representing a series of length l of d-dimensional vectors) is calculated by Equation (27) using these Q, K, and V. Here, inside Equation (27), masking by the matrix D (a square matrix of l rows and l columns) is performed. The element D of the n-th row and m-th column of the matrix D nm is shown in Equation (28). As shown in Equation (28), when performing the learning of the retentive network, when the matrix D acts as a mask, when estimating the n-th output in the series, the m-th input in the series is taken into consideration based on the value of D nm . The value of that D nm is γ n-m when n ≥ m, or 0 when n < m. That is, in order to estimate the n-th output, the coefficient 0 is multiplied by the m-th input where n < m. That is, during learning, the information of the m-th input where n < m is not used. On the other hand, in order to estimate the n-th output, the coefficient γ n-m is multiplied by the m-th input where n ≥ m. Here, as the value of γ, 0 < γ < 1.

[0013] That is, in the retentive network described in Non-Patent Document 6, when estimating the output value at a certain position, the state when performing a process earlier in the series has a smaller influence. In other words, in the retentive network described in Non-Patent Document 6, the internal state at a certain position in the series becomes more difficult to remember as the series progresses. That is, the memory of the state decays according to γ n-m .

[0014] Here, the problem in the retentive network described in Non-Patent Document 6 is that the matrix element D represented by Equation (28) nm has a fixed value of γ when n ≥ m n-m . That is, there is a problem that the value of the matrix element D nm cannot be flexibly changed according to the situation (for example, according to the input data) when n ≥ m

[0015] The present invention has been made based on the above recognition of problems, and attempts to expand the retentive network so that the value of D in Equation (28) in the prior art can be flexibly changed according to the situation (for example, according to the input data). That is, the present invention attempts to provide an extended retentive network device, an inference device, and a program equipped with such an extended retentive network nm . [Means for Solving the Problems] <s

[0016] [1] To solve the above problems, an extended retentive network device according to an aspect of the present invention inputs data X, which is a matrix representing a series of lengths l (l is a positive integer) of d-dimensional (d is a positive integer) vectors, and estimates data Y, which is a matrix representing a series of lengths l of d-dimensional vectors, based on the data X. The extended retentive network device has an extended retentive network unit configured to extend the retentive network. The extended retentive network unit has machine-learnable internal parameters. When adjusting the internal parameters by machine learning, the extended retentive network unit calculates the n-th (where 1 ≤ n ≤ l) d-dimensional vector in the series of the data Y by applying a predetermined coefficient D nm to the m-th (where 1 ≤ m ≤ l) d-dimensional vector in the series of the data X. When n < m, D nm = 0, and when n = m, D nm= 1, and if n > m, then 0 ≤ D nm <1, and for a given n, the larger the value of (nm), the greater the D nm The value of is small, and for a given n, when n>m, the value of D for m=n-1 nm From the value of m=1, D nm The degree of the sequential attenuation up to the value of X depends on the data X.

[0017] [2] Also, an extended retentive network device according to one aspect of the present invention is an extended retentive network device including an extended retentive network unit having a network configured by extending a retentive network, which inputs data X, which is a matrix representing a sequence of a d-dimensional (d is a positive integer) vector of length l (l is a positive integer), and estimates data Y, which is a matrix representing a sequence of a d-dimensional vector of length l, based on the data X. The extended retentive network unit has internal parameters that can be machine-learned, and when the data X is represented by a matrix of l rows and d columns and the data Y is represented by a matrix of l rows and d columns, when adjusting the internal parameters by machine learning, the extended retentive network unit calculates the data Y by performing operations represented by Equations (33), (34), (35), (36), (37), (38), and (39) (these formulas are described in detail later) (wherein W Q and W K What is d row d K Column(d K is a matrix of positive integers, and W V is a d-by-d matrix, and Θ and Θ(bar) are l-by-d K is the column matrix, Θ n is the nth row of Θ, Θ(bar) is the complex conjugate of Θ, and W r is a d-by-1 matrix, f() is a positive function, r is an l-by-1 matrix, and r n is the nth row element of r, γ<0, and W Q and W K and W V and W rγ and θ are learnable internal parameters), and when performing sequential inference using the internal parameters obtained by machine learning, the extended retentive network unit calculates the data Y by performing the calculations expressed by equations (40) and (41) (these equations will be described in detail later) (where S n is the internal state corresponding to the nth state of the sequence, and K n is the nth row of K, and V n is the nth row of V, and Q n is the nth row of Q, and Y n is the nth row of Y), an extended retentive network device.

[0018] [3] Furthermore, an inference device according to one aspect of the present invention comprises an extended retentive network device according to [1] or [2] above, and a feedforward network unit having a second internal parameter and configured as a neural network, wherein the feedforward network unit inputs data based on the data Y output from the extended retentive network unit of the extended retentive network device, and outputs data calculated by processing the feedforward network based on the value of the internal parameter.

[0019] [4] Another aspect of the present invention is a program for causing a computer to function as the extended retentive network device of [1] or [2] above.

[0020] [5] Another aspect of the present invention is a program for causing a computer to function as the inference device described in [3] above. [Effects of the Invention]

[0021] According to the present invention, the conventional retentive network is extended to enable more flexible modeling for sequence prediction, which is expected to improve accuracy. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a block diagram showing a schematic functional configuration of an extended retentive network device according to an embodiment of the present invention; [Figure 2] 10 is a block diagram showing a schematic functional configuration of an inference device using an extended retentive network unit of the extended retentive network device according to the embodiment. FIG. [Figure 3] 10 is a schematic diagram showing data dependency during learning of the extended retentive network unit according to the embodiment. FIG. [Figure 4] 10 is a schematic diagram showing data dependency during sequential inference of the extended retentive network unit according to the embodiment. FIG. [Figure 5] FIG. 2 is a block diagram showing an example of the internal configuration of a computer for realizing the device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0023] Next, an embodiment of the present invention will be described with reference to the drawings.

[0024] To facilitate understanding of the embodiment, the configuration of the conventional technology and its problems will be described below in order, and then the configuration of the retentive network device according to the embodiment will be described with reference to FIG. 1. The embodiment uses a machine learning technique. The embodiment can be realized using, for example, neural network technology. The data series to be processed in the embodiment is typically time-series data, but is not necessarily limited to time-series data and may be other types of series of data.

[0025] A recurrent neural network (RNN), as described in Non-Patent Document 1, is a general term for neural networks that have internal recurrent cyclic connections. A recurrent neural network can retain an internal state (memory) that summarizes an input sequence received as a series into a vector with a fixed number of dimensions, regardless of the length of the sequence. This allows a neural network incorporating a recurrent neural network to behave dynamically based on the memory of inputs that have been input in the past.

[0026] Non-Patent Document 2 describes an Elman network, which is a type of recurrent neural network and has the most basic configuration. The Elman network has a network configuration expressed by the following formulas (1) and (2).

[0027]

number

[0028]

number

[0029] However, in the above formula, x t is the input vector to the network at discrete time t. Also, h t is the hidden state vector at time t. t corresponds to the memory summarizing the input sequence up to time t. t is the output vector from this network at time t. Also, W h , U h , b h , W y , and b y is a learnable parameter of this network. Also, σ h and σ y represents some activation function.

[0030] Recurrent neural networks can be trained using a technique commonly known as backpropagation through time (BPTT). In other words, in BPTT, the backpropagation method commonly used in training neural networks is applied to a network with a recurrent structure deployed along the time axis. Specifically, in BPTT, the values ​​of the trainable parameters are first initialized with some random numbers. Then, input sequences (training examples) are sequentially fed to the network to produce output sequences. This output sequence is then used to solve some downstream task, and the error is then backpropagated. Training is achieved by repeating this backpropagation parameter update for multiple training examples.

[0031] However, in practice, it is known that simple recurrent neural networks such as those illustrated above cannot stably learn tasks that require long time series dependencies, despite their potential time series memory ability.

[0032] For example, in the above Elman network, the hidden state vector h at a certain time t is t is expressed in the form of equation (3) below.

[0033]

number

[0034] In other words, x seen from time t t-a For information on (U h ) a (Actually, the activation function is inserted between each multiplication.) Therefore, in many cases, there is a problem that errors backpropagated to distant times during training diverge or disappear.

[0035] Non-Patent Document 3 describes Long Short-term Memory (LSTM) as one of the means for solving the above problem. Non-Patent Document 4 describes Gated Recurrent Unit (GRU) as one of the means for solving the above problem. LSTM and GRU are recurrent neural networks with a gate mechanism.

[0036] For example, in the case of a typical LSTM, the network configuration is expressed by the following equations (4) to (9).

[0037]

number

[0038]

number

[0039]

number

[0040]

number

[0041]

number

[0042]

number

[0043] However, in these equations, x t is the input vector to this network at time t. Also, h t is the hidden state and output vector at time t. Also, c tis the cell state vector. f , U f , b f , W i , U i , b i , W o , U o , b o , W c , U c , and b c is a learnable parameter of this network. Also, σ g is a sigmoid function, and σ c is a hyperbolic function.

[0044] Also,

number

[0045] In this network, the cell state vector c t This c t x included in t-a The information is in the form of equation (11) below.

[0046]

number

[0047] That is, x expressed by equation (11) t-a Regarding information, the problem of divergence / disappearance as in the case of the above formula (3) is unlikely to occur. However, even these recurrent neural networks with gate mechanisms are known to be insufficient in their ability to memorize very long sequences.

[0048] On the other hand, a neural network called a Transformer, described in Non-Patent Document 5, has an internal mechanism called attention. While a recurrent neural network retains memory that summarizes input sequences into vectors with a fixed number of dimensions regardless of the sequence length, the attention mechanism instead retains all inputs as they are. This enables the Transformer to model the output with high performance.

[0049] There are several variations of attention mechanisms, one of which is a mechanism called self-attention. Similar to recurrent neural networks, self-attention is used for tasks that require only a single sequence as input and calculates the same number of outputs as the input. The network configuration of self-attention is expressed by the following equations (12) to (15). Equation (15) represents an operation called self-attention operation.

[0050]

number

[0051]

number

[0052]

number

[0053]

number

[0054] X is a matrix representing the input sequence to this self-attention mechanism. 1 is the sequence length. d is the number of dimensions of the input vector. Y is a matrix representing the output sequence. W Q , WK , and W V is a learnable parameter. Also, the "T" attached to the upper right of K in equation (15) represents the transpose of the matrix. The size of each matrix is ​​as shown in equations (16) to (19) below.

[0055]

number

[0056]

number

[0057]

number

[0058]

number

[0059] The softmax function used in equation (15) performs calculations according to equation (20) below.

[0060]

number

[0061] Transformers generally use a configuration called multi-head attention. In multi-head attention, the input vector is divided into multiple parts (each part is called a head), and the self-attention calculation described above is performed independently for each head. The values ​​of the learnable parameters are also stored for each head. The calculation results of each head are then combined, and finally, the output is calculated through some kind of neural network mapping.

[0062] A sequence model defined by a recurrent neural network or the like is called an "autoregressive model", and in the inference at each time, the output is calculated from the memory of the input before that time. Such a sequence model is generally used for the purpose of predicting the future such as the next input. On the other hand, by definition, in the self-attention operation, the sequence is input in a batch, and the output is calculated while referring to all the inputs before and after the sequence for each input vector of the sequence. This form is called non-causal self-attention. There is a method to realize an autoregressive model by masking so as not to refer to the input sequence after that position in the operation for each input vector of the sequence so as to be equivalent to a recurrent neural network. This method is called causal self-attention.

[0063] The masking in the above causal self-attention is represented by the following equations (21) and (22).

[0064]

Equation

[0065]

Equation

[0066] That is, the difference between Equation (21) and the above Equation (15) is that the mask matrix M is applied in the operation at the stage before the softmax function. The value of the n-th row and m-th column of the matrix M is M represented by Equation (22). nm That is, by multiplying the matrix elements in the region where n < m by 0, it is made not to refer to the input sequence at the subsequent position.

[0067] When using an autoregressive model with causal self-attention for sequential inference, the entire input sequence is retained and the self-attention calculation described above is performed for each time. In other words, at each time, one input is received and one output is obtained. The sequence length increases by one as time progresses. The end of the resulting output sequence is then used as the output vector for that time.

[0068] Autoregressive models using the transformer self-attention mechanism have been proposed as neural networks with higher modeling performance than recurrent neural networks and are used in many applications. However, as mentioned above, the self-attention mechanism needs to retain the entire input sequence as is. Therefore, when training the self-attention mechanism, it is advantageous to use matrix operations instead of BPTT (backpropagation through time) calculations as in recurrent neural networks. However, using the self-attention mechanism for sequential inference is inefficient in terms of computational complexity and memory capacity.

[0069] In other words, BPTT (backpropagation through time) calculations require computation time proportional to the sequence length. On the other hand, the matrix operations of the self-attention mechanism can be calculated independently for each time point of the sequence, so the computation time can be reduced when parallel computation is performed. However, when using the self-attention mechanism for sequential inference, the computational complexity requires computations using the entire sequence as input at each time point, and memory capacity is required to store the entire input sequence.

[0070] Non-Patent Document 6 describes the technology of a Retentive Network as one of the proposals for solving the above problems. A Retentive Network is sometimes written as "RetNet." A Retentive Network has a mechanism called retention, which serves as an alternative to self-attention. The retention mechanism is capable of parallel computation during learning, just like self-attention learning. On the other hand, during sequential inference, the retention mechanism can perform calculations using a fixed-dimensional memory, just like a recurrent neural network, by using the parameters learned by parallel computation as they are. In other words, the retention mechanism solves the above-mentioned problems of the self-attention mechanism.

[0071] The configuration of the retentive network during learning is expressed by the following equations (23) to (28).

[0072]

number

[0073]

number

[0074]

number

[0075]

number

[0076]

number

[0077]

number

[0078] Also, the configuration during inference of the retentive network is represented by the following equations (29) and (30).

[0079]

Number

[0080]

Number

[0081] Here, X, W Q , W K , and W V are the same as those in the attention mechanism. γ and θ are learnable parameters. In Non-Patent Document 6, γ is a constant, but generally γ is not limited to a constant. Regarding γ, 0 < γ < 1. That is, the state S n at the n-th position in the sequence depends on the state S m at the m-th position in the sequence as shown in Equation (29) (where m < n), but as the difference between m and n increases, the influence of the state S m on the state S n decays. Θ is a matrix with complex numbers as elements, and Θ (bar) is the complex conjugate of Θ. Θ n is the n-th column component of the matrix Θ. That is as shown in Equation (31).

[0082]

Number

[0083] Re() in Equations (27) and (30) represents the action of extracting the real component of a complex number.

[0084] Also, here W K may be a complex matrix. In exchange for an increase in the number of parameters, performance improvement can be expected.

[0085] Also, Q in equations (28) and (29) n , K. n , and V n are the components of Q, K, and V corresponding to time n. Also, S in Equations (29) and (30) n represents the internal state (memory) at time n (i.e., at the nth time in the sequence), where 0<γ<1.

[0086] The retentive network can also have a multi-head configuration similar to the transformer. When using a multi-head configuration, Y (or Y n Instead of extracting the real component when calculating Y (or Y n ) can be combined as complex numbers, and when some mapping is performed to obtain the final output, the mapping can be converted from complex numbers to real numbers (the real components can be extracted).

[0087] Retention mechanisms are generally used in the same way as self-attention mechanisms in transformers. That is, a retention mechanism and a feedforward layer (generally a multilayer perceptron with internal nonlinear activation) are serially connected, and each layer is further connected with a skip connection (adding the input value of that part to the output value), forming a single layer. The entire network is then constructed by further connecting these multiple layers. However, the usage is not necessarily limited to this.

[0088] [Issues to be resolved] The retention calculation of the conventional method, the Retentive Network, performs sequential updates of the internal state (memory), similar to recurrent neural networks and LSTMs. The sequential update of the internal state (memory) is expressed by the above equation (29). In other words, the Retention Network has the same performance as the Transformer, while resolving the inefficiency of sequential inference.

[0089] On the other hand, the sequential update of the internal state of the LSTM is expressed by the following equation (32).

[0090]

number

[0091] The comparison between the retentive network and the LSTM is as follows: As shown in equation (32), the LSTM uses a variable f t and i t In contrast, in the retention operation, the new state is calculated by mixing the previous state and the input using the weights γ·e iθ is a constant that does not depend on the input. In other words, retention operations have the problem of lacking flexibility in state updates.

[0092] The retentive network device according to this embodiment aims to solve the above problems by using the theory of discretization of continuous-time state space models. The present embodiment, which will be described below, improves the expressive power of the model and enables more advanced sequence modeling.

[0093] In the retainive network device of this embodiment, the retention mechanism according to the conventional technology is expanded to have the following network configuration.

[0094] The network configuration at the time of learning in the retentive network device of this embodiment is expressed by the following equations (33) to (39).

[0095]

number

[0096]

number

[0097]

number

[0098]

number

[0099]

number

[0100]

number

[0101]

number

[0102] The network configuration during sequential inference in the retentive network device of this embodiment is expressed by the following equations (40) and (41). n , Y n , K. n , V n , and Q n are the data of the n-th row of X, Y, K, V, and Q, respectively.

[0103]

number

[0104]

number

[0105] However, X, W Q , W K , W V, γ, and θ are the same as those in the retention mechanism expressed by equations (23) to (28). That is, X is a matrix representing the input sequence to the extended retention mechanism of this embodiment. Also, W Q , W K , and W V is a learnable parameter. γ and θ are also learnable parameters. W r is a learnable parameter. r is expressed in equation (42) below.

[0106]

number

[0107] The function f() is any appropriately selected positive function. For example, the function f() may be an exponential function or a softplus function. The softplus function is a smoothly increasing function that returns a positive value close to 0 if the input value is less than or equal to 0, and returns a value close to the input value if the input value is greater than 0. r is a matrix with l rows and 1 column. r n is the nth element of r, where e is Napier's constant and i is the imaginary unit.

[0108] That is, during sequential inference, X n (Data in the nth row of matrix X (d-dimensional vector)) is input sequentially starting from n=1. The retentive network device inputs X, which is the nth (n=1, 2, . . .) input data in the series. n When is input, state S n-1 and X n and based on state S n and output data Y n (Data in the nth row of matrix Y (d-dimensional vector)) State S n is the nth input data X n is the internal state of the retentive network device as a result of inputting n is the first input data X1 to the nth input data Xn where S0 is the initial state of the retentive network device.

[0109] If the sequence is a time sequence, the nth sequence corresponds to time n.

[0110] That is, as shown in equation (40), the state S n is the state S n-1 (1st term) and X n Based on K n T V n (2nd term) is calculated by adding them together with a weighting factor. n And, as shown in equation (36), r n depends on the input data X.

[0111] 1 is a block diagram showing a schematic functional configuration of an extended retentive network device according to this embodiment. As shown in the figure, the extended retentive network device 1 is configured to include an extended retentive network unit 10. The extended retentive network unit 10 also has a storage unit 17.

[0112] The extended retentive network device 1 of this embodiment replaces the retention function of the retentive network according to the conventional technology. All the configurations proposed in the retentive network according to the conventional technology except for the retention mechanism are inherited. Also, as mentioned above, the extended retentive network device 1 can generally be used as a multi-head configuration.

[0113] The functions of the extended retentive network device 1 can be realized, for example, by a computer and a program. The extended retentive network device 1 may also have internal storage means as needed. The storage means may be, for example, a program variable or a memory allocated by the execution of the program. Non-volatile storage means such as a magnetic hard disk drive or a solid-state drive (SSD) may also be used as needed. The computer for realizing the extended retentive network device 1 is not necessarily limited to a computer with a von Neumann architecture. It may also include a configuration with a non-von Neumann architecture required to realize the extended retentive network device 1. At least some of the functions of the extended retentive network device 1 may be realized as a dedicated electronic circuit rather than a program.

[0114] In the extended retentive network device 1, sequence data is inferred from front to back. Here, "front" and "back" are defined as follows. That is, the extended retentive network device 1 calculates the nth internal state and the nth output data based on the nth input data of the sequence and the (n-1)th internal state. Here, the side with the smaller number (the n-1th side) is the "front" in the sequence, and the side with the larger number (the nth side) is the "back" in the sequence.

[0115] The extended retentive network unit 10 is realized using a neural network. In other words, the values ​​of the internal parameters of the extended retentive network unit 10 can be adjusted by machine learning. When learning the internal parameters, the extended retentive network unit 10 calculates data Y by performing the operations expressed by the above formulas (33) to (39). Furthermore, when sequential inference is performed, the extended retentive network unit 10 calculates the state S by performing the operations expressed by the above formulas (40) and (41). n (1≦n≦l) and data Y n (Also, calculate 1≦n≦l).

[0116] The storage unit 17 stores the above state S n (1≦n≦l) is sequentially stored in the storage unit 17. The storage unit 17 may also store the initial state S0. That is, the storage unit 17 holds the state at each point in time during sequential inference.

[0117] 2 is a block diagram showing a schematic functional configuration of an inference device using the above-described extended retentive network unit 10. As shown in the figure, the inference device 2 is configured with the extended retentive network unit 10, a feedforward network unit 20, and addition units 31 and 32.

[0118] The extended retentive network unit 10 has the same configuration and functions as those described with reference to Fig. 1. Note that a configuration including this extended retentive network unit 10 may be considered as the extended retentive network device 1.

[0119] The feedforward network unit 20 is configured using a neural network and calculates an output value based on an input value. The internal parameters of the feedforward network unit 20 can be learned by machine learning.

[0120] The adder 31 adds two pieces of input data and outputs the result. The two pieces of input data input to the adder 31 may be data with dimensions. That is, the two pieces of input data input to the adder 31 may be matrices or vectors of the same size. In the configuration shown in the figure, specifically, the adder 31 adds X, which is input data to the inference device 2, and Y, which is data output from the extended retentive network unit 10.

[0121] Similar to the above-described adder unit 31, the adder unit 32 also adds two pieces of input data and outputs the result. The two pieces of input data input to the adder unit 32 may be data with dimensions. That is, the two pieces of input data input to the adder unit 32 may be matrices or vectors of the same size. In the illustrated configuration, specifically, the adder unit 32 adds the data output from the adder unit 31 and the data output from the feedforward network unit 20.

[0122] 2, the inference device 2 operates as follows. That is, the extended retentive network unit 10 performs a calculation based on input data X and outputs data Y. The adder unit 31 adds the input data X and Y, which is the output from the extended retentive network unit 10. The adder unit 31 passes the result of the addition to the feedforward network unit 20 and the adder unit 32. The feedforward network unit 20 performs a calculation using internal parameter values ​​at that time based on the data passed from the adder unit 31, and outputs the result. The adder unit 32 adds the data passed from the adder unit 31 and the data output from the feedforward network unit 20. The output from the adder unit 32 is the output of the inference device 2.

[0123] During learning, the inference device 2 adjusts the values ​​of the internal parameters of the feedforward network unit 20 and the extended retentive network unit 10 using the backpropagation method based on the loss between the output data from the adder unit 32 and the given correct answer data. During inference, the inference device 2 performs calculations based on the values ​​of the internal parameters obtained by learning, and outputs the inference result (output from the adder unit 32) to the outside.

[0124] Next, data dependencies in the extended retentive network unit 10 will be described with reference to Figures 3 and 4. The network configuration in the extended retentive network unit 10 realizes these dependencies. However, the physical network configuration or the procedure of calculations by the program do not necessarily have to be as shown in Figures 3 and 4, and an equivalent configuration or calculation procedure may be used.

[0125] 3 is a schematic diagram showing the data dependency during learning of the extended retentive network unit 10. In the figure, data X, W r , W Q , W K , W V , r, Q, K, V, θ, Θ, D, γ, and Y correspond to the respective symbols in the above formula. r , W Q , W K , W V , θ, and γ) are internal parameters that can be learned. As shown, r is the parameter W r and the input data X. Q is calculated based on the parameter W Q and the input data X. K is calculated based on the parameter W K and the input data X. V is calculated based on the parameter W V and input data X. Θ is calculated based on parameters θ and r. D is calculated based on Θ and r. D is calculated based on Θ, r and parameter γ. Output data Y is calculated based on D, γ, r, Q, K, and V.

[0126] The feature of the extended retentive network unit 10 according to this embodiment is that the retentive network of the prior art is extended as follows. That is, in this embodiment, when learning, input data X and parameters W rIn other words, r depends on the input data X. Θ, D, and output data Y are calculated based on r. The output data Y is calculated based on D, γ, r, Q, K, and V.

[0127] FIG. 4 is a schematic diagram showing the data dependency relationship during sequential inference in the extended retentive network unit 10. In the figure, data X n , W r , W Q , W K , W V , r n , Q n , K. n , V n ,θ,γ,S n-1 , S n , and Y n The symbols correspond to the symbols in the above formula. r , W Q , W K , W V , θ, and γ) are internal parameters that can be learned. n is the parameter W r and input data X n It is calculated based on Q n is the parameter W Q and input data X n It is calculated based on K n is the parameter W K and input data X n It is calculated based on V n is the parameter W V and input data X n It is calculated based on the following. n is the previous state S n-1 and,r n And, K. n And, V n The output data Y is calculated based on θ and γ. n Q n , and S n It is calculated based on the following.

[0128] The feature of the extended retentive network unit 10 according to this embodiment is that it extends the conventional retentive network as follows. That is, in this embodiment, during learning, the input data X n and the parameter W r are used to calculate r n . That is, r n depends on the input data X n . The state S n is calculated based on such r n . The output data Y n is calculated based on S n and Q n .

[0129] That is, the extended retentive network device 1 described while referring to FIG. 1 is as follows. That is, the extended retentive network device 1 inputs data X, which is a matrix representing a series of vectors of length l (l is a positive integer) in d dimensions (d is a positive integer), and estimates data Y, which is a matrix representing a series of vectors of length l in d dimensions, based on the data X. The extended retentive network device 1 has an extended retentive network unit 10 having a network configured to extend a retentive network. The extended retentive network unit 10 has machine-learnable internal parameters and is configured to perform the following operations when adjusting the internal parameters by machine learning. The extended retentive network unit 10 calculates the n-th (where 1 ≤ n ≤ l) d-dimensional vector in the series of the data Y by applying a predetermined coefficient D nm to the m-th (where 1 ≤ m ≤ l) d-dimensional vector in the series of the data X. When n < m, D nm = 0. When n = m, D nm = 1. When n > m, 0 ≤ D nm < 1, and the value of D nm is smaller as the value of (n − m) is larger for a given n. When n > m, for a given n, when m = n - 1, Dnm From the value of m=1, D nm The degree of the sequential attenuation up to the value of is configured to depend on the data X.

[0130] The extended retentive network device 1 described with reference to Fig. 1 is as follows. That is, the extended retentive network device 1 includes an extended retentive network unit having a network configured by extending a retentive network, which inputs data X, which is a matrix representing a sequence of d-dimensional (d is a positive integer) vectors with length l (l is a positive integer), and estimates data Y, which is a matrix representing a sequence of d-dimensional vectors with length l, based on the data X. The extended retentive network unit 10 has internal parameters that can be machine-learned, and when the data X is represented by a matrix of l rows and d columns and the data Y is represented by a matrix of l rows and d columns, when adjusting the internal parameters by machine learning, calculates the data Y by performing operations represented by equations (33), (34), (35), (36), (37), (38), and (39) (these equations are specifically described above) (wherein W Q and W K What is d row d K Column(d K is a matrix of positive integers, and W V is a d-by-d matrix, and Θ and Θ(bar) are l-by-d K is the column matrix, Θ n is the nth row of Θ, Θ(bar) is the complex conjugate of Θ, and W r is a d-by-1 matrix, f() is a positive function, r is an l-by-1 matrix, and r n is the nth row element of r, γ<0, and W Q and W K and W V and W r γ and θ are internal parameters that can be learned). When performing sequential inference using the internal parameters obtained by machine learning, the extended retentive network unit 10 calculates the data Y by performing the calculations expressed by equations (40) and (41) (these equations are specifically described above) (where S n is the internal state corresponding to the nth state of the sequence, and K n is the nth row of K, and V n is the nth row of V, and Q n is the nth row of Q, and Y n is the nth row of Y).

[0131] 2, the inference device 2 is as follows: That is, the inference device 2 includes the above-described extended retentive network device 1 and a feedforward network unit having a second internal parameter and configured by a neural network, wherein the feedforward network unit receives data based on the data Y output from the extended retentive network unit of the extended retentive network device, and outputs data calculated by processing the feedforward network based on the value of the internal parameter.

[0132] FIG. 5 is a block diagram showing an example of the internal configuration of a computer for implementing the above-described embodiment. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element in the RAM 902 has an address and can be accessed using the address. RAM stands for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port 903 via the bus 906.

[0133] Note that the computer for implementing the embodiment is not limited to a computer with a von Neumann architecture, and may include a special hardware configuration suitable for a neural network.

[0134] The extended retentive network device 1 in the above-described embodiment can implement at least some of the functions of the inference device 2 using a computer and a program. In this case, the program for implementing this function may be recorded on a computer-readable recording medium and then loaded and executed by a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases. The program may also be designed to implement some of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system.

[0135] The method for realizing a predetermined function using machine learning techniques is as follows. Here, for convenience, a unit of function that can be machine-learned is referred to as a "function unit." The function unit calculates data to be output based on input data. The function unit uses internal parameter values ​​when calculating the data to be output. The internal parameters can be updated and optimized through a learning process. When training the function unit, training data is used. The training data includes input data to the function unit and a correct answer for output data calculated based on the input data. During the learning process, the function unit reads the input data included in the training data and references output data based on the input data and using the internal parameter values ​​at that time. This output data is an estimated value calculated based on the internal parameters at that time and does not necessarily coincide with the correct answer. To update the internal parameters, the difference between the estimated value calculated and output by the function unit based on the input data at that time and the correct answer corresponding to the input data is calculated. This difference is called an error, loss, etc. The difference calculated here may be, for example, the absolute value of the difference between scalars, a squared error, a cross-entropy error, or a difference calculated by other methods. Based on the calculated difference, the internal parameter values ​​can be updated using backpropagation. This operation adjusts the internal parameter values ​​so as to reduce the error. By performing the above operation multiple times (multiple times) using a predetermined amount (large amount) of training data, the internal parameter values ​​are optimized. In other words, the functional unit is adjusted to perform the processing exemplified by the training data provided. After a sufficient amount of training has been performed, the training process may be terminated. The set of trained internal parameter values ​​at this time may be stored in a storage device or the like, enabling estimation based on the training results. A function that enables data input and output, including the internal parameters, may be called a "model." In other words, a model includes internal parameters. Training the functional unit may be called "model training."One way to implement the model is to use a neural network, and depending on the type of processing to be performed, an appropriate form of neural network can be used.

[0136] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Industrial Applicability]

[0137] The present invention can be used, for example, in modeling and inference of sequence data, although the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0138] 1. Extended Retentive Network Device 2 Reasoning device 10. Extended Retentive Network Section 17 Memory section 20 Feedforward Network Section 31,32 Addition section 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. an extended retentive network unit having a network configured by extending a retentive network, which inputs data X, which is a matrix representing a sequence of d-dimensional (d is a positive integer) vectors with length l (l is a positive integer), and estimates data Y, which is a matrix representing a sequence of d-dimensional vectors with length l based on the data X; An extended retentive network device comprising: The extended retentive network unit has internal parameters that can be machine-learned, When adjusting the internal parameters by machine learning, The extended retentive network unit applies a predetermined coefficient D to the m-th (where 1≦m≦l) d-dimensional vector in the sequence of the data X. ｎｍ By applying the above formula, the n-th (where 1≦n≦l) d-dimensional vector in the series of data Y is calculated, If n<m, then D ｎｍ = 0, When n = m, D ｎｍ = 1, When n>m, 0≦D ｎｍ <1, and the larger the value of (n-m) for a given n, the greater the D ｎｍ The value of is small, For n>m, for a given n, D for m=n-1 ｎｍ From the value of m = 1, ｎｍ The degree of sequential attenuation to the value of depends on the data X. Enhanced Retentive Network Device.

2. an extended retentive network unit having a network configured by extending a retentive network, which inputs data X, which is a matrix representing a sequence of d-dimensional (d is a positive integer) vectors with length l (l is a positive integer), and estimates data Y, which is a matrix representing a sequence of d-dimensional vectors with length l based on the data X; An extended retentive network device comprising: The extended retentive network unit has internal parameters that can be machine-learned, The data X is expressed as a matrix of l rows and d columns, When the data Y is expressed as a matrix of l rows and d columns, When the extended retentive network unit adjusts the internal parameters by machine learning, [Equation 1] [Equation 2] [Equation 3] [Equation 4] [Equation 5] [Equation 6] [Equation 7] The data Y is calculated by performing the following calculation (where W Ｑ and W Ｋ That is d row d Ｋ Column (d Ｋ is a matrix of positive integers, W Ｖ is a matrix with d rows and d columns, Θ and Θ (bar) are in the l row d Ｋ is a column matrix, Θ ｎ is the nth row of Θ, Θ (bar) is the complex conjugate of Θ W ｒ is a d-row, 1-column matrix, f() is a positive-valued function, r is a matrix with l rows and 1 column, r ｎ is the nth row element of r, γ<0, W Ｑ and W Ｋ and W Ｖ and W ｒ γ and θ are internal parameters that can be learned), When the extended retentive network unit performs sequential inference using the internal parameters obtained by machine learning, [Equation 8] [Equation 9] The data Y is calculated by performing the following calculation (where: S ｎ is the internal state corresponding to the nth state of the sequence, K ｎ is the n-th row of K, V ｎ is the nth row of V, Q ｎ is the nth row of Q, Y ｎ is the nth row of Y), Enhanced Retentive Network Device.

3. an extended retentive network device according to claim 1 or 2; a feedforward network unit having a second internal parameter and configured as a neural network; Equipped with The feedforward network unit inputs data based on the data Y output from the extended retentive network unit of the extended retentive network device, and outputs data calculated by processing the feedforward network based on the value of the internal parameter. Reasoning device.

4. A program for causing a computer to function as the extended retentive network device according to claim 1 or 2.

5. A program for causing a computer to function as the inference device according to claim 3.

Images