Method, system and computer readable storage medium for processing long input sequence

Abstract

The invention relates to a method, a system and a computer readable storage medium for processing a long input sequence. A method for processing a long input sequence, comprising: receiving a long input sequence; based on the long input sequence, an adjusted long input sequence is generated using a state memory playback module, where the state memory playback module is configured to perform the following: perform a convolution operation on the long input sequence to obtain an adjustment factor, the adjustment factor comprises information of a time period for tracing back the convolution kernel length associated with the convolution operation from the current moment; and applying the adjustment factor to the long input sequence, thereby obtaining an adjusted long input sequence; and generating an output result using a state-space model based on the adjusted long input sequence.

Description

Technical Field

[0001] This disclosure relates generally to data processing, and more specifically to methods, systems, and computer-readable storage media for processing long input sequences. Background Technology

[0002] With the increasing demand for computers to process long input sequences, long sequence modeling has attracted widespread interest. Architectures based on state space models (SSMs) have demonstrated superior performance in long sequence modeling due to their linear computational complexity and excellent ability to capture long-range dependencies.

[0003] However, taking the S4 model as an example, since the sampling step size of long input sequences is not exactly the same or uniform, but the S4 model uses the same time step size for each time interval, the model introduces errors when processing long input sequences, and these errors accumulate continuously as long input sequences are processed. The S5 and S6 (Mamba) models mainly rely on recursive structures to address the different sampling step size issues. The S5 model introduces a learnable step size for each time step (position step), while the S6 model introduces data-dependent parameter settings. However, in the recursive structure, inference computation is limited to parallel scanning, which severely impacts training efficiency and creates a heavier inference burden when processing long inputs at once.

[0004] Long input sequence processing techniques that need improvement at at least one of the above-mentioned levels are needed in this field. Summary of the Invention

[0005] It should be understood that the general description above and the detailed description below are exemplary and illustrative, and are intended to provide further explanation of the present disclosure as described in the claims.

[0006] One aspect of this disclosure provides a method for processing a long input sequence, comprising: receiving the long input sequence; generating an adjusted long input sequence based on the long input sequence using a state memory replay module, wherein the state memory replay module is configured to perform the following operations: performing a convolution operation on the long input sequence to obtain an adjustment factor, wherein the adjustment factor includes information on a time period tracing back from the current time to the length of the convolution kernel associated with the convolution operation; applying the adjustment factor to the long input sequence to obtain the adjusted long input sequence; and generating an output result based on the adjusted long input sequence using a state space model.

[0007] As described above, the adjustment factor is determined as follows: in response to determining that the state memory replay module is applied to a linear layer, a second convolution operation is performed on the padded long input sequence, and a linear operation is performed on the result of the second convolution operation, the result of which is used to determine the adjustment factor.

[0008] The adjustment factor is determined as follows: in response to determining that the state memory replay module should not be applied to the linear layer, a third convolution operation is performed on the padded long input sequence, and the result of the third convolution operation is used to determine the adjustment factor.

[0009] As described above, the state memory replay module is used to: apply an activation function to the adjustment factor and multiply it element-wise with the long input sequence to obtain the adjusted long input sequence.

[0010] As described above, the convolution operation is a one-dimensional convolution operation.

[0011] As described above, the long input sequence includes one of the following: text, image, speech, video, and mathematical expression.

[0012] As described above, the long input sequence has a non-uniform sampling step size.

[0013] The method described above is used for autoregressive language modeling or long sequence dependency modeling.

[0014] Another aspect of this disclosure provides a system for processing long input sequences, comprising: a processor configured to perform the method as described in any of the preceding methods.

[0015] Another aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0016] The method and system disclosed herein improve the ability of state-space models to process long input sequences and increase the accuracy of output results. Attached Figure Description

[0017] Various embodiments of this disclosure are described in conjunction with the accompanying drawings.

[0018] Figures 1A-1D The diagram illustrates the states and inputs with and without disturbance.

[0019] Figure 2 This is a schematic diagram illustrating the adjustment of the sampling step according to some embodiments of this disclosure.

[0020] Figure 3This is a flowchart of a method for processing long input sequences according to some embodiments of the present disclosure.

[0021] Figure 4 This is a flowchart of a process executed by a state memory replay module according to some embodiments of the present disclosure.

[0022] Figure 5 This is a flowchart of a process executed by a state memory replay module according to some embodiments of the present disclosure.

[0023] Figure 6 This is a schematic diagram of a combination of a state space model and a state memory replay module according to some embodiments of this disclosure.

[0024] Figures 7A-7D This is a comparison chart of experimental results for the S4 model based on some embodiments of this disclosure.

[0025] Figure 8 This is a block diagram of a system for processing long input sequences according to some embodiments of the present disclosure.

[0026] Figure 9 This is a block diagram of a computer-readable storage medium according to some embodiments of the present disclosure.

[0027] Figure 10 This is a block diagram of a computer program product according to some embodiments of the present disclosure. Detailed Implementation

[0028] Embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings, but the disclosure is not limited thereto but is defined solely by the claims. In the drawings, some elements may be enlarged and drawn out of scale for illustrative purposes. Wherever possible, the same reference numerals will be used in all drawings to denote the same or similar parts.

[0029] Although the terminology used in this disclosure is selected from commonly known and used terms, some terms mentioned in this disclosure may have been chosen by the applicant in his or her judgment, and their detailed meanings are explained in the relevant sections of the description herein. Furthermore, this disclosure should be understood not only by the actual terms used, but also by the meaning implied by each term.

[0030] Numerous specific details are set forth in the description provided herein. However, it should be understood that embodiments of this disclosure may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this disclosure.

[0031] In this application, ordinal numbers such as "first," "second," and "third" are used to distinguish different instances of objects with the same name. The ordinal numbers "first," "second," and "third" do not indicate a relative order of the indicated objects in time, space, sequence, or other aspects.

[0032] Regarding the basic concepts of state-space models, a state-space model is defined by the following equations:

[0033]

[0034] y(t)=Cx(t)+Du(t) (2)

[0035] in, Let m represent an input sequence of dimension m. Indicates a potential state.

[0036] The S4 model constructs a set of structured state-space sequence model parameters for each dimension of the input sequence to build a Single Input Single Output (SISO) system, that is, for the input... Broadcast the same set of parameters to each dimension Specifically, the S4 model uses a bilinear approach to perform discretization:

[0037]

[0038] in, Since matrix D can be viewed as a residual join, it is omitted here. For each element u (p) Let p∈1,2,...m, and t be a fixed discretization step, and each step is the same. Therefore, S4 is a function with trainable parameters. The parameterized model of Δt. Assuming x0 = 0, we can obtain:

[0039]

[0040] Therefore, convolution can be used. To calculate the output, where

[0041]

[0042] is the convolution kernel, and L is the length of the input sequence. Using the Normal Plus Low Rank (NPLR) parameterization method employed by the S4 model, the convolution computation in the S4 model... The operation is performed in one operation. However, because the S4 model uses a uniform time step for each time interval, it will produce errors when the input sequence has irregularly sampled data.

[0043] The S5 model introduces a learnable step size at each time step to handle irregularly sampled data. Furthermore, the S5 model extends from SISO systems to Multiple Input Multiple Output (MIMO) systems. Specifically, the S5 model diagonalizes the SSM dynamics, transforming the matrix... Reparameterize into a focus matrix, and Configured as a matrix, instead of the vectorization used in the S4 model. and However, the varying stride introduced by the S5 model for different time steps constrains the efficiency of the SSM convolution operations. Although the S5 model utilizes a diagonalized state transition matrix, its computational complexity remains O(mL). Furthermore, the SSM parameters of both the S4 and S5 models are fixed after training, thus the SSM parameters are independent of the input sequence data, limiting the model's flexibility.

[0044] The S6 model introduces data-dependent parameter settings and uses trainable linear layers to map the input to each step. The S6 model improves flexibility by introducing a time step Δt and transforming the parameters in the time-invariant systems of the S4 and S5 models into time-varying parameters. However, because its parameters are time-dependent, the S6 model cannot efficiently perform SSM computations using convolutions, resulting in slower training speeds while maintaining a computational complexity of O(mL).

[0045] Event-triggered control (ETC) theory is a dynamic control strategy that emphasizes updating control signals only when specific events occur. The following explanation, based on ETC theory, illustrates the errors caused by SSM (System-Solved Mode) when processing irregularly sampled data, also known as the steady-state problem. To ensure the system remains stable, trigger events are used to sample the input control signal; the sampling points t1, t2, ... need to meet specific requirements. Typically, Lyapunov functions can be used. To assess stability, outside the stable point, it monotonically decreases, reaching a minimum of 0 at the stable point. The choice that produces... The sampling points with decreasing trends are used to ensure system stability. Returning to the linear system described in equation (1), assume that the input control signal satisfies the linear relationship u(t) = Tx(t), where Then equation (1) becomes:

[0046]

[0047] It is easy to conclude It is a Lyapunov function, where the symmetric positive definite matrix is... To maintain, the following conditions must be met.

[0048]

[0049] Where M∈R n×n It is also a symmetric positive definite matrix.

[0050] The actual sampled input u(t) i At the sampling point The sampling error is expressed as:

[0051]

[0052] Therefore, equation (7) can be rewritten as:

[0053]

[0054] right Differentiation yields:

[0055]

[0056] Therefore, by setting t0 = 0, the following triggering conditions can be obtained to ensure system stability:

[0057]

[0058] Where κ∈(0,1) is an optional constant, e(t) - Let represent the left-hand limit of the error e at point t. That is, a new control signal is input before the system becomes unstable. Therefore, the obtained sampled input control sequence ensures the exponential stability of the system:

[0059]

[0060] Where ι is a positive constant.

[0061] Assuming a one-dimensional input is provided, the selected parameters are... A simulation experiment was conducted on the system with a time window of [0, 10] and a time grid width of 0.01. Figures 1A-1D This diagram illustrates the states and inputs with and without disturbance. Figure 1AAs shown, u represents the input, x1 and x2 represent the states, and the gray dashed line represents the trigger time. Based on the sampled input obtained from ETC theory, the system state eventually reaches stability.

[0062] Further perturbations are added to the sampled data points, with the perturbations confined within the time grid width, such as... Figure 1B As shown, u' represents the scrambled input, and the perturbation is almost unobservable. Figure 1C This shows the state and input without any disturbance. Figure 1D The diagram illustrates the state and input with added perturbation. It shows that the system maintains stability without perturbation, but stability is not guaranteed after perturbation. This demonstrates that inconsistency between the actual sampling points and the desired sampling grid leads to instability. Therefore, for the SSM model, irregularly sampled data can cause significant numerical errors.

[0063] Therefore, it can be concluded that for a specific SSM model with fixed parameters, when the input does not meet the stability condition, it will lead to an unstable state problem. As sampling errors propagate along with the hidden state, numerical errors will result.

[0064] Therefore, according to one aspect of this disclosure, a method for processing long input sequences is provided.

[0065] Assuming an input perturbation ε at the sampling point, we obtain For a small perturbation ε(t), u′(t) = u(t) + ε(t), therefore the observed state is expressed as z(t) = x(t) - e(t), and the difference between the observed state and the actual state is the error e(t) = x(t) - z(t). Using the Lyapunov function as an indicator of the stability of the observed error in the system, the smaller the absolute value of e(t), the smaller the impact of noise and uncertainty on the system performance. Therefore, for the input replay factor... The input to be adjusted is u adj (t)=h τ (t)u(t), where z(t) is the state value of the observer. For Lyapunov functions We can obtain:

[0066]

[0067] in P is a positive definite symmetric matrix. It is a function with fixed coefficients determined by the SSM parameters.

[0068] Therefore, it can be concluded that for the input controller h τ Applying additional constraints can improve the convergence of the system, especially when h τ (·)≡1 (corresponding to the S4 model), ||h τ ||∞ =1. To make the control factor h... τ Incorporating information from the time interval [t-τ, t], embodiments of this disclosure envision a convolution operation utilizing a convolution kernel of length τ. Further utilize the activation function σ to ensure By utilizing these operations, system stability is improved. Therefore, this disclosure proposes a State Memory Replay (SMR) module, which utilizes convolutional learnable variables to contain multiple input states in a long input sequence, i.e., learnable memory, to achieve adaptive learning and optimization of long input sequences.

[0069] Based on the above explanation, it can be understood that automatic adjustment of long input sequences can be achieved by using learnable variables containing multiple input states of a long input sequence to control how the sampling information is used.

[0070] Figure 2 This is a schematic diagram illustrating the adjustment of the sampling step according to some embodiments of this disclosure.

[0071] exist Figure 2 On the left, the real signal is a long input sequence of irregular samples indicated by solid circles. By adjusting the sampling steps of the real signal using a state memory replay module, a sampling grid indicated by hollow circles and its corresponding adjusted signal are obtained. Figure 2 On the right, below u1 all the way to u L Each sampled data point in the long input sequence is represented by an adjustment factor Δ, which is obtained by performing convolution and activation functions on the long input sequence in the state memory replay module.

[0072] Therefore, the state memory replay module according to some embodiments of this disclosure enhances the SSM model by utilizing multiple memory steps through convolutional learnable variables that include multiple input steps. After a long input sequence is processed by the state memory replay module of an embodiment of this disclosure, the SSM model can be represented as:

[0073]

[0074] Where τ represents the kernel length, σ Sig (·) represents the Sigmoid function.

[0075] Figure 3 This is a flowchart of a method 300 for processing long input sequences according to some embodiments of the present disclosure.

[0076] At step S310, a long input sequence is received.

[0077] At step S320, an adjusted long input sequence is generated based on the long input sequence using the state memory replay module.

[0078] At step S330, the output result is generated based on the adjusted long input sequence and using a state-space model.

[0079] In some embodiments, a long input sequence may include one of the following: text, image, speech, video, or mathematical expression.

[0080] In some embodiments, long input sequences may have non-uniform sampling step sizes.

[0081] In some embodiments, method 300 can be used for autoregressive language modeling or long sequence dependency modeling.

[0082] Figure 4 This is a flowchart of a process 400 performed by a state memory replay module according to some embodiments of the present disclosure. In some embodiments, process 400 may be performed in step S320 of method 300. In some embodiments, process 400 may be performed independently of method 300.

[0083] At step S402, a convolution operation is performed on the long input sequence to obtain an adjustment factor. The adjustment factor includes information about the time period from the current moment back to the length of the convolution kernel associated with the convolution operation.

[0084] At step S404, an adjustment factor is applied to the long input sequence to obtain an adjusted long input sequence.

[0085] Therefore, the state memory replay module according to some embodiments of this disclosure performs convolution operations on a long input sequence. The convolution operation targets multiple input steps of historical moments related to the current moment in the long input sequence, so that the adjustment factor is obtained based on multiple memory steps, thereby enabling adjustment of the data at the current moment based on the state memory. Thus, the state memory replay module according to some embodiments of this disclosure achieves efficient computation of the state space model through convolution operations, improving the adaptability of the state space model to long input sequences with different sampling intervals.

[0086] Although the above describes the current sampling point of the long input sequence at the current moment, it should be understood that the current moment is not limited to describing the sampling point corresponding to a specific time in the long input sequence, but can also describe the sampling point corresponding to a specific position in the long input sequence, such as information about a specific position in a long text input sequence.

[0087] In some embodiments, the convolution operation can be a one-dimensional convolution operation.

[0088] In some embodiments, step S404 may include applying an activation function to the adjustment factor and multiplying it element-wise with the long input sequence to obtain an adjusted long input sequence.

[0089] Figure 5 This is a flowchart of a process 500 performed by a state memory replay module according to some embodiments of the present disclosure. In some embodiments, process 500 may be performed in step S402 of process 400. In some embodiments, process 500 may be performed independently of process 400.

[0090] In step S502, it is determined whether the state memory replay module is applied to the linear layer.

[0091] At step S504, in response to the determination that the state memory replay module is applied to the linear layer, a second convolution operation is performed on the padded long input sequence, and a linear operation is performed on the result of the second convolution operation. The result of the linear operation is used to determine the adjustment factor.

[0092] At step S506, in response to determining that the state memory replay module should not be applied to the linear layer, a third convolution operation is performed on the padded long input sequence, and the result of the third convolution operation is used to determine the adjustment factor.

[0093] The following shows the code of a state memory replay module according to some embodiments of this disclosure:

[0094]

[0095]

[0096] The code above is just an example of a state memory replay module, and the scope of this invention is not limited thereto.

[0097] Figure 6 This is a schematic diagram of a combination of a state space model and a state memory replay module according to some embodiments of this disclosure. Figure 6 The diagrams illustrate combinations of the S4, S5, S6, SPADE, and Mega models with the state memory replay module. By processing long input sequences using the state memory replay module, the data at the current moment is adjusted based on the state memory before being provided to the state space model, thereby improving the accuracy of the state space model in processing long input sequences.

[0098] Figures 7A-7D This is a comparison chart of experimental results for the S4 model according to some embodiments of this disclosure. The sequence modeling task involves sampling 100 equidistant points from the function sin(5πt) as input, then adding a perturbation to the input to slightly change the sampling points, and finally using the S4 model to predict the perturbated input. Figure 7A and Figure 7B The prediction results of the S4 model without and with the state memory replay module are shown respectively. When there is a perturbation in the input, the prediction results of the S4 model with the state memory replay module are more consistent with the input. Figure 7C and Figure 7D The table shows the total capacity of the potential states and their absolute values ​​when the S4 model is used without and with the state memory replay module. For both the original and scrambled states, the total capacity of the absolute values ​​of the potential states in the S4 model is greater than 200. However, when the S4 model uses the state memory replay module, the total capacity of the absolute values ​​of the potential states is less than 10, which means that the potential states have reached stability. This implies that the problem of unstable states in the S4 model has been solved.

[0099] Table 1 shows the analysis performed on the Pendulum dataset using a state-space model and a state memory replay module. This dataset has irregular sampling points and varying sampling intervals. The pattern represents the computational pattern of the state-space model, and MSE (Mean Squared Error) represents the mean squared error calculated on the test set. As shown in Table 1, the S4 and S5 models show a significant reduction in error using the state memory replay module, and the errors of other models are also reduced accordingly. Therefore, the state memory replay module can improve the performance of state-space models.

[0100] Table 1: Test results of the state-space model on the pendulum dataset

[0101]

[0102] Table 2 presents the analysis performed on the WikiText-103 dataset using state-space models and the state memory replay module, and compares the results with those of Transformer and Transformer-LS models to evaluate the performance for autoregressive language modeling. PPL (perplexity) represents the perplexity score, and the left and right sides of the " / " represent the results with and without the state memory replay module, respectively. As shown in Table 2, the state memory replay module reduces perplexity for all models, and the SPADE model even outperforms the state-of-the-art Transformer-LS when using it. Therefore, the state memory replay module is suitable for improving the performance of state-space models for autoregressive language modeling.

[0103] Table 2: Test results of the state-space model on the WikiText-103 dataset

[0104]

[0105] Table 3 shows the analysis performed on the Long Range Arena (LRA) benchmark task using the state-space model and the state memory replay module, and compares the results with those of the Transformer and Transformer-LS models to evaluate the performance for modeling long-range dependencies. The tasks include byte-level text classification, ListOps (for mathematical expressions), byte-level document retrieval, sequence CIFAR-10 (for images), and path discovery. The left and right sides of the " / " indicate the results with and without the state memory replay module, respectively. As shown in Table 3, the state memory replay module improves the score for all models. Therefore, the state memory replay module is suitable for improving the performance of state-space models for modeling long-range dependencies.

[0106] Table 3: Test results of the state-space model in the remote arena

[0107]

[0108] Table 4 shows a comparison of training speeds using the state-space model and the state memory replay module on the WikiText-103 dataset. The analysis uses the S6 model implemented in torch as the baseline (1.0 × speed), showing the speed ratios of other models relative to the S6 model. Table 4 shows that using the state memory replay module does not significantly reduce the training speed of the state-space model, and the relative speed relationships between different state-space models remain unchanged. Therefore, the state memory replay module enhances the computational power of the state-space model without affecting training efficiency.

[0109] Table 4: Training speed of the state-space model on the WikiText-103 dataset

[0110]

[0111] Therefore, the combination of the state space model and the state memory replay module according to some embodiments of this disclosure can alleviate the unstable state problem, enhance the generalization ability of the state space model to irregular sampling points, and improve the modeling performance of long input sequences.

[0112] According to another aspect of this disclosure, a system for processing long input sequences is provided.

[0113] Figure 8This is a block diagram of a system 800 for processing long input sequences according to some embodiments of the present disclosure. System 800 includes a processor 802. Processor 802 may include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and various other processing units or cores (e.g., arithmetic logic unit, integer unit, floating-point unit, tensor unit, ray tracing core, etc.). Processor 802 is configured to perform the operations described above. Figures 3-5 The methods described.

[0114] According to another aspect of this disclosure, a computer-readable storage medium is provided.

[0115] Figure 9 This is a block diagram of a computer-readable storage medium 900 according to some embodiments of the present disclosure.

[0116] A computer-readable storage medium 900 stores a computer program 950. When executed by a processor, the computer program 950 implements the above-mentioned combination. Figures 3-5 The steps of each method are described.

[0117] According to another aspect of this disclosure, a computer program product is provided.

[0118] Figure 10 This is a block diagram of a computer program product 1000 according to some embodiments of the present disclosure.

[0119] Computer program product 1000 may include computer program 950. Computer program 950, when executed by a processor, implements the above-mentioned... Figures 3-5 The steps of each method are described.

[0120] Embodiments of this disclosure have been described with reference to the accompanying drawings. These embodiments are illustrative and not restrictive.

Claims

1. A method for processing long input sequences, characterized in that, include: Receive the long input sequence; Based on the long input sequence, an adjusted long input sequence is generated using a state memory replay module, wherein the state memory replay module is used to perform the following operations: A convolution operation is performed on the long input sequence to obtain an adjustment factor, wherein the adjustment factor includes information about the time period from the current moment back to the convolution kernel length associated with the convolution operation; and The adjustment factor is applied to the long input sequence to obtain the adjusted long input sequence; and The output is generated using a state-space model based on the adjusted long input sequence.

2. The method as described in claim 1, characterized in that, The adjustment factor is determined in the following manner: In response to determining that the state memory replay module is applied to a linear layer, a second convolution operation is performed on the padded long input sequence, and a linear operation is performed on the result of the second convolution operation, the result of which is used to determine the adjustment factor.

3. The method as described in claim 1, characterized in that, The adjustment factor is determined in the following manner: In response to determining that the state memory replay module should not be applied to the linear layer, a third convolution operation is performed on the padded long input sequence, the result of which is used to determine the adjustment factor.

4. The method as described in claim 1, characterized in that, The state memory replay module is used for: An activation function is applied to the adjustment factor and multiplied element-wise with the long input sequence to obtain the adjusted long input sequence.

5. The method as described in claim 1, characterized in that, The convolution operation is a one-dimensional convolution operation.

6. The method as described in claim 1, characterized in that, The long input sequence includes one of the following: text, image, speech, video, and mathematical expression.

7. The method as described in claim 1, characterized in that, The long input sequence has a non-uniform sampling step size.

8. The method as described in claim 1, characterized in that, The method is used for autoregressive language modeling or long sequence dependency modeling.

9. A system for processing long input sequences, characterized in that, include: A processor configured to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-8.

Images