Small-sample long-time-sequence prediction method and system based on global attention and local attention

Abstract

The invention provides a small-sample long-time-sequence prediction method and system based on global attention and local attention, and the method comprises the steps: firstly, employing the global attention, and carrying out the fast Fourier transform of a long sequence and a short sequence, so as to extract the periodicity and corresponding number features; then, selecting a similar convolution method to carry out local attention so as to reveal association between different paragraphs in the short sequence and the auxiliary long sequence; and finally, combining the two attention mechanisms to obtain a final weight, and combining the feedforward neural network with the residual error to obtain output. According to the method, the similarity between the short sequence and the auxiliary long sequence is effectively mined, so that accurate small sample prediction is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of time series analysis, and in particular to a small sample long time series prediction method and system based on global attention and local attention. Background Art

[0002] Time series forecasting has a wide range of applications in many areas, including predicting weather conditions, conducting economic analysis, and managing traffic congestion. In real-world scenarios, we often face the challenge of small-sample forecasting, such as data completion from a single sensor, prediction of complex physical phenomena (such as turbulence), and aftershock detection. The motivation for small-sample forecasting stems from a unique challenge inherent in time series analysis. The goal is to predict the future of newly collected short series by leveraging information from multiple long time series. This problem essentially involves effectively transferring temporal information from long series to new short series (target domain) where training data is scarce. A characteristic of this scenario is that the training data available in the target domain is small, so it is classified as a small-sample forecasting problem. This problem is common in various contexts, such as estimating the long-term revenue of a newly opened store branch, or predicting the future traffic flow of a newly built parking lot or intersection. The core goal is to identify similarities and potential dependencies between established long and short series and make predictions based on them.

[0003] The few-shot prediction problem involves transferring knowledge from the source domain to the target domain, where the training set of the target contains only a limited number of samples. Common approaches to solving few-shot time series prediction currently include fine-tuning-based transfer learning, meta-feature-based meta-models, and sequential latent variable models (SLVMs). Transfer learning is trained on a data-rich task and then fine-tuned to transfer to a data-scarce few-shot task. It has been widely used for few-shot prediction, but requires retraining on the real task. Meta-learning in time series involves selecting a suitable prediction model based on few-shot features (such as standard deviation and mean), providing a general approach to few-shot prediction models, but requires understanding the meta-features of the samples. On the other hand, SLVMs mine internal information by extracting latent states rather than simple autoregression, but most are self-iterative and tend to accumulate errors. Many such few-shot prediction models tend to directly apply features learned from long sequences to short sequences without considering the possible dependency inconsistencies between long and short sequences. This direct feature transfer method is usually not practical, and the performance in solving the challenge of few-shot time series prediction is mostly unsatisfactory.

[0004] Patent document CN116523109A discloses a long time series prediction method and system based on sequence decomposition and subsequence attention mechanism, including: step 1: traverse the original sequence by sliding window, and divide it into three component sequences of equal length, namely: solve the mean of the sequence of multiple time points in the past as the trend term component at the current moment, solve the standard deviation as the scale term component, and standardize the original sequence according to the mean and standard deviation as the stationary term component; step 2: based on the attention mechanism, calculate the probability of random events between any two subsequences, and obtain the prediction result of each subsequence for the future; step 3: weight and synthesize all prediction results through a linear screening mechanism to obtain the final prediction result. However, this patent cannot completely solve the existing technical problems and cannot meet the needs of the present invention. Summary of the invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a small sample long time series prediction method and system based on global attention and local attention.

[0006] The small sample long time series prediction method based on global attention and local attention provided by the present invention includes:

[0007] Step 1: Use the global attention mechanism to input a complete time series, decompose the time series into components across various frequencies through Fourier transform, perform Fourier transform on all auxiliary long sequences to obtain the Fourier transformed matrix, and then perform Fourier transform on the short sequence; downsample the Fourier transformed matrix and align it with the Fourier transformed short sequence; learn weights through the global attention mechanism to obtain the similarity between the short sequence and the auxiliary long sequence in terms of overall fluctuation;

[0008] Step 2: Adopt the local attention mechanism to randomly select a segment in each auxiliary long sequence to represent the overall features of the auxiliary long sequence, and extract the similarity between the short sequence and different segments in the auxiliary long sequence;

[0009] Step 3: Combine the global attention mechanism with the local attention mechanism to get the final weight, and obtain the prediction result by combining the feedforward neural network with the residual.

[0010] Preferably, first, input the source domain auxiliary long sequence X = [x1, x2, x3, ..., x N ]∈R N×T and target domain short sequence Where t0<<T, N represents the total number of samples; t0 represents the length of the short sequence in the target domain; T represents the length of the auxiliary long sequence in the input source domain; R N×T Represents the space where the auxiliary long sequence of the input source domain is located; Represents the space where the short sequence of the target domain is located;

[0011] Perform fast Fourier transform on the source domain auxiliary long sequence and the target domain short sequence to obtain the corresponding frequency domain representation and Downsample X(f) to x ω (f) The corresponding frequency, get the downsampled matrix X s (f);

[0012] Then, the weight A is calculated through the global attention mechanism global , and perform Monte Carlo sampling on X to obtain a random sampling sequence X R , using x ω As the convolution kernel, for X R Perform convolution and calculate the local attention weight A local ;

[0013] Finally, the global and local attention results are combined to obtain the weighted attention weight of the auxiliary long sequence segment: V = A global ×A local , the final prediction output is obtained by superimposing the final attention weight and residual

[0014] Preferably, the step 1 comprises:

[0015] The distance between the short sequence and the cluster center is measured by the global attention mechanism, and the expression is:

[0016] D(x ω ,C)=dist(x ω ,C)

[0017] Among them, x ω =[x1,x2,x3,…,xt] represents a short sequence, C = [c1,c2,c3,…,ct] represents the cluster center mapped to the short sequence space;

[0018] Then obtain the mapping of the cluster center of the long sequence in the short sequence space, and use Fourier transform to map the long and short sequences to their respective Fourier spaces. The energy of the time series in the time domain is equal to the energy expressed in the frequency domain. The expression is:

[0019]

[0020] Where t is the time variable, x(t) represents the signal in the time domain; f is the frequency variable, x(f) is the Fourier transform of x(t), representing the signal in the frequency domain;

[0021] Downsample the long sequence and select the frequency components that exist in the Fourier transform of the short sequence. The expression is:

[0022]

[0023] Among them, x(n) is the time domain signal, X(k) is the frequency domain signal, and N is the total number of samples;

[0024] Take every α values ​​from the long sequence and get the corresponding frequency value in the Fourier space of the short sequence. The expression is:

[0025] y[m]=x[m·α]

[0026] Where x[m·α] is the original long sequence, y[m] is the downsampled short sequence, m is the index in the short sequence, and α is the ratio of the long sequence length to the short sequence length;

[0027] The attention weight is calculated from the short sequence matrix randomly sampled from the auxiliary long sequence, and the expression is:

[0028]

[0029] Among them, Q is the matrix used to calculate the attention weights, representing the query; K is both the key and value matrix; is the scaling factor of the dot product, d k is the dimension of the key;

[0030] Attention(Q,K) represents the attention weights calculated using the query matrix Q and the key matrix K, which is used to capture the correlation between different positions in the sequence.

[0031] Preferably, step 2 comprises:

[0032] The local attention mechanism is used to capture the similarity between short sequences and segments of the same length in long sequences. The expression is:

[0033]

[0034] Where N is the total number of samples, k is the lag step size; ρ k represents the autocorrelation coefficient when the sample lag step is k, which measures the similarity of the signal at different lag times; X t represents the signal value at time t; represents the mean value of the signal;

[0035] For a single fragment:

[0036]

[0037] Among them, P represents the probability that a single sample is not sampled, λ represents the number of segments sampled each time, M represents the total number of segments, and C represents the number of sampling times; therefore, the expected number of sampling segments is expressed as:

[0038]

[0039] When K is large enough, it is considered to be sufficient to reflect the sample characteristics; at the same time, for most time series data that follow a normal distribution:

[0040]

[0041] Among them, z α / 2 is the quantile of the normal distribution, σ is the sample standard deviation, θ is the mean, D is the total number of samples, and CI represents the confidence interval;

[0042] Local attention performs linear transformation on long and short sequences:

[0043] F′=θ(F)

[0044] G′=Φ(G)

[0045] The similarity is calculated through the convolution mechanism:

[0046] C=F′*G′

[0047] Among them, F is the matrix obtained by Monte Carlo sampling, G is the input short sequence, C represents the calculated similarity matrix, Φ and Θ represent linear transformations, and F′ and G′ are matrices after linear transformation.

[0048] Preferably, the step 3 comprises:

[0049] First, by means of matrix multiplication, the global attention and local attention are combined to obtain the final weighted result of attention, which comprehensively reflects the input target domain short sequence x ω The similarity with each auxiliary long sequence in the source domain is expressed as:

[0050] V=A global ×A local

[0051] Then, the obtained attention matrix V is transformed accordingly through the feedforward neural network, and the expression is:

[0052] FFN(V)=σ(W2σ(W1V+b1)+b2)

[0053] Wherein, W1 and W2 represent the weight matrices of the feedforward neural network respectively; b1 and b2 represent the bias vectors of the feedforward neural network respectively;

[0054] A residual connection is introduced in the prediction output to add the original input of the short sequence to the output of FFN. The expression is:

[0055]

[0056] The small sample long time series prediction system based on global attention and local attention provided by the present invention includes:

[0057] Module M1: Using the global attention mechanism, a complete time series is input, and the time series is decomposed into components across various frequencies through Fourier transform. All auxiliary long sequences are Fourier transformed to obtain the Fourier transformed matrix, and then the short sequence is Fourier transformed; the matrix after Fourier transform is downsampled and aligned with the short sequence after Fourier transform; the weights are learned through the global attention mechanism to obtain the similarity between the short sequence and the auxiliary long sequence in terms of overall fluctuation;

[0058] Module M2: adopts the local attention mechanism to randomly select a segment in each auxiliary long sequence to represent the overall features of the auxiliary long sequence, and extracts the similarity between the short sequence and different segments in the auxiliary long sequence;

[0059] Module M3: Combine the global attention mechanism with the local attention mechanism to obtain the final weight, and obtain the prediction result by combining the feedforward neural network with the residual.

[0060] Preferably, first, input the source domain auxiliary long sequence X = [x1, x2, x3, ..., x N ]∈R N×T and target domain short sequence Where t0<<T, N represents the total number of samples; t0 represents the length of the short sequence in the target domain; T represents the length of the auxiliary long sequence in the input source domain; R N×T Represents the space where the auxiliary long sequence of the input source domain is located; Represents the space where the short sequence of the target domain is located;

[0061] Perform fast Fourier transform on the source domain auxiliary long sequence and the target domain short sequence to obtain the corresponding frequency domain representation and Downsample X(f) to x ω (f) The corresponding frequency, get the downsampled matrix X s (f);

[0062] Then, the weight A is calculated through the global attention mechanism global , and perform Monte Carlo sampling on X to obtain a random sampling sequence X R , using x ω As the convolution kernel, for X R Perform convolution and calculate the local attention weight A local ;

[0063] Finally, the global and local attention results are combined to obtain the weighted attention weight of the auxiliary long sequence segment: V = A global ×Alocal , the final prediction output is obtained by superimposing the final attention weight and residual

[0064] Preferably, the module M1 comprises:

[0065] The distance between the short sequence and the cluster center is measured by the global attention mechanism, and the expression is:

[0066] D(x ω ,C)=dist(x ω ,C)

[0067] Among them, x ω =[x1,x2,x3,…,xt] represents a short sequence, C = [c1,c2,c3,…,ct] represents the cluster center mapped to the short sequence space;

[0068] Then obtain the mapping of the cluster center of the long sequence in the short sequence space, and use Fourier transform to map the long and short sequences to their respective Fourier spaces. The energy of the time series in the time domain is equal to the energy expressed in the frequency domain. The expression is:

[0069]

[0070] Where t is the time variable, x(t) represents the signal in the time domain; f is the frequency variable, x(f) is the Fourier transform of x(t), representing the signal in the frequency domain;

[0071] Downsample the long sequence and select the frequency components that exist in the Fourier transform of the short sequence. The expression is:

[0072]

[0073] Among them, x(n) is the time domain signal, X(k) is the frequency domain signal, and N is the total number of samples;

[0074] Take every α values ​​from the long sequence and get the corresponding frequency value in the Fourier space of the short sequence. The expression is:

[0075] y[m]=x[m·α]

[0076] Where x[m·α] is the original long sequence, y[m] is the downsampled short sequence, m is the index in the short sequence, and α is the ratio of the long sequence length to the short sequence length;

[0077] The attention weight is calculated from the short sequence matrix randomly sampled from the auxiliary long sequence, and the expression is:

[0078]

[0079] Among them, Q is the matrix used to calculate the attention weights, representing the query; K is both the key and value matrix; is the scaling factor of the dot product, d k is the dimension of the key;

[0080] Attention(Q,K) represents the attention weights calculated using the query matrix Q and the key matrix K, which is used to capture the correlation between different positions in the sequence.

[0081] Preferably, the module M2 comprises:

[0082] The local attention mechanism is used to capture the similarity between short sequences and segments of the same length in long sequences. The expression is:

[0083]

[0084] Where N is the total number of samples, k is the lag step size; ρ k represents the autocorrelation coefficient when the sample lag step is k, which measures the similarity of the signal at different lag times; X t represents the signal value at time t; represents the mean value of the signal;

[0085] For a single fragment:

[0086]

[0087] Among them, P represents the probability that a single sample is not sampled, λ represents the number of segments sampled each time, M represents the total number of segments, and C represents the number of sampling times; therefore, the expected number of sampling segments is expressed as:

[0088]

[0089] When K is large enough, it is considered to be sufficient to reflect the sample characteristics; at the same time, for most time series data that follow a normal distribution:

[0090]

[0091] Among them, z α / 2 is the quantile of the normal distribution, σ is the sample standard deviation, θ is the mean, D is the total number of samples, and CI represents the confidence interval;

[0092] Local attention performs linear transformation on long and short sequences:

[0093] F′=θ(F)

[0094] G′=Φ(G)

[0095] The similarity is calculated through the convolution mechanism:

[0096] C=F′*G′

[0097] Among them, F is the matrix obtained by Monte Carlo sampling, G is the input short sequence, C represents the calculated similarity matrix, Φ and Θ represent linear transformations, and F′ and G′ are matrices after linear transformation.

[0098] Preferably, the module M3 comprises:

[0099] First, by means of matrix multiplication, the global attention and local attention are combined to obtain the final weighted result of attention, which comprehensively reflects the input target domain short sequence x ω The similarity with each auxiliary long sequence in the source domain is expressed as:

[0100] V=A global ×A local

[0101] Then, the obtained attention matrix V is transformed accordingly through the feedforward neural network, and the expression is:

[0102] FFN(V)=σ(W2σ(W1V+b1)+b2)

[0103] Wherein, W1 and W2 represent the weight matrices of the feedforward neural network respectively; b1 and b2 represent the bias vectors of the feedforward neural network respectively;

[0104] A residual connection is introduced in the prediction output to add the original input of the short sequence to the output of FFN. The expression is:

[0105]

[0106] Compared with the prior art, the present invention has the following beneficial effects:

[0107] The present invention proposes a small sample prediction method based on global attention and local attention mechanisms, which greatly improves the accuracy and efficiency of long time series prediction tasks; compared with other models, the present invention uses two attention mechanisms to derive relevant weights, and then combines them to obtain the final weights, and integrates them with the residual through a feedforward neural network to obtain the output, which solves the problem of how to learn the potential dependency between short sequences and auxiliary long sequences, and greatly improves the accuracy and interpretability of predictions in the field of small sample prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0109] Figure 1 It is the overall framework diagram of the algorithm of the present invention;

[0110] Figure 2 Schematic diagram of the global attention framework during training;

[0111] Figure 3 Schematic diagram of the global attention framework during prediction;

[0112] Figure 4 Schematic diagram of the local attention framework. DETAILED DESCRIPTION

[0113] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0114] Example

[0115] The present invention first designs a distribution-aware sequence decomposition algorithm. The main calculation mode of the algorithm is: solving the mean of the sequence of the past L time points as the trend term component at the current moment, solving the standard deviation as the scale term component, and standardizing the original sequence according to the mean and standard deviation as the stationary term component. The process traverses the entire sequence in a sliding window manner and divides the original sequence into three component sequences of equal length.

[0116] The present invention finds that when solving the problem of small sample time series prediction, traditional models usually transfer the features learned from long sequences directly to the target short sequence through forward neural networks or recursive neural networks, but fail to establish a strong association between the support set and the target. Therefore, they cannot fully consider the potential relationship between long sequences and short sequences, resulting in poor performance.

[0117] To effectively address this problem and capture the relationship between long and short time series, we derived two key insights: 1) Spatial dependence: Data from individual sensors exhibit consistent fluctuation patterns. Therefore, sensors with similar characteristics can significantly improve the accuracy of mutual predictions. 2) Temporal dependence: If the initial parts of two series are similar, then their subsequent parts will also have certain similarities. These observations highlight the importance of understanding spatiotemporal relationships in accurately making small-sample time series forecasts.

[0118] Based on the above insights, the present invention first designs a global attention mechanism algorithm that captures global dependencies. The main calculation method of the algorithm is: fast Fourier transform of long sequences and short sequences, and fast Fourier transform (FFT) of long sequences and short sequences to extract periodicity and corresponding quantitative features. Then, we select the long sequence components that match the short sequence frequency for improved transformation.

[0119] The main role of global attention is to measure the overall similarity between the target domain short sequence and the source domain long sequence. Specifically, in our previous data processing, we assumed that the sequence segments belonging to the same parking lot exhibit consistent fluctuation characteristics. From a mathematical point of view, data sampled from the same source can naturally be clustered into one category. Therefore, global attention effectively measures the distance between short sequences and cluster centers through the attention mechanism.

[0120] Subsequently, the present invention designs a local attention mechanism algorithm that captures local dependencies to evaluate the similarity between subsequences and short sequences within a long sequence. Based on our previous assumptions, the present invention believes that if the initial parts of two sequences are similar, then their subsequent parts will also have certain similarities. Therefore, the present invention focuses on the similarity between the short sequence of the target domain and the subsequences within each long sequence of the source domain, and predicts future fluctuations based on this. Inspired by the calculation of the autocorrelation coefficient, we chose a convolution-like method for local attention. After linear transformation, the input short sequence is regarded as a convolution kernel, which is convolved on the long sequence segment matrix obtained by Monte Carlo sampling to obtain weights. This technique aims to discover the correlation between short sequences and auxiliary long sequence segments.

[0121] After establishing the above basic assumptions and methods, we invented a small-shot prediction model based on a dual attention mechanism. Initially, the model uses two attention mechanisms to derive the relevant weights. They are then combined to obtain the final weights, and integrated with the residual through a feedforward neural network to obtain the output. This approach effectively mines the similarities between short sequences and auxiliary long sequences, thereby achieving accurate small-shot prediction.

[0122] Overall structure

[0123] This algorithm mainly consists of two parts, namely the global attention module and the local attention module. The global attention measures the similarity between the short sequence and the entire auxiliary long sequence in the frequency domain, and the local attention is used to extract the similarity between the short sequence and each subsequence fragment of the auxiliary long sequence.

[0124] The first part is the global attention module. A complete time series is input, and the Fourier transform decomposes the time series into components across various frequencies, which helps to analyze periodic and trend-related correlations. All auxiliary long sequences are Fourier transformed to obtain the Fourier transformed matrix, and then the short sequence is Fourier transformed and the Fourier transformed matrix is ​​downsampled and aligned with the short sequence. Then, the optimized attention mechanism is used to learn the weights to focus on finding the similarities between the short sequence and the auxiliary long sequence in terms of overall fluctuations.

[0125] The second part is the local attention module. The local attention extracts the similarity between short sequences and long sequence segments, and represents the overall characteristics of the long sequence by randomly selecting a segment in each sequence. Figure 4 As shown in the figure, local attention uses the optimized convolutional attention mechanism, and the model recognizes the similarities between short and long sequence segments. Combining the Monte Carlo algorithm and the convolution method, the model greatly reduces the time and space complexity and improves efficiency.

[0126] Finally, the model integrates the extracted global and local attention and generates the final output through a feedforward neural network. The model combines global spatial dependency, local temporal dependency, and correlation between variables to make small sample predictions richer and more reliable. In addition, the model performs direct multi-step (DMS) predictions, avoiding the error accumulation of iterative predictions and achieving high reliability of long-term predictions.

[0127] Basic process

[0128] like Figure 1 As shown, first, input the source domain auxiliary long sequence X = [x1, x2, x3, ..., x N ]∈R N×T and target domain short sequence Where t<<T. Perform fast Fourier transform on long and short sequences to obtain the corresponding frequency domain representation and Downsample X(f) to x ω (f) The corresponding frequency, get the downsampled matrix X s (f). Then, the weight A is calculated through the global attention mechanism global At the same time, Monte Carlo sampling is performed on X to obtain a random sampling sequence X R , using x ω As the convolution kernel, for X R Perform convolution and calculate the local attention weight A local . Combining the global and local attention results, we get the weighted attention weight of the auxiliary long sequence segment: V = A global ×A local , the final prediction output is obtained by superimposing the final attention weight and residual

[0129] Each module process

[0130] Global Spatial Attention Module

[0131] Global attention in the model architecture refers to learning the similarity between the overall periodicity and fluctuation characteristics of the short sequence and the auxiliary long sequence. In our previous data processing, we assumed that the sequence segments belonging to the same parking lot exhibit consistent fluctuation characteristics. From a mathematical point of view, data sampled from the same source can naturally be clustered into one category. Therefore, global attention effectively measures the distance between the short sequence and the cluster center through the attention mechanism.

[0132] D(x ω ,C)=dist(x ω ,C)

[0133] where x ω =[x1,x2,x3,…,xt] represents the short sequence, and C = [c1,c2,c3,…,ct] represents the cluster center mapped to the short sequence space. We measure the distance through the attention mechanism, and now we need to find the mapping of the cluster center of the long sequence in the short sequence space. Inspired by the frequency domain MLP model, our invention considers using Fourier transform to map long and short sequences to their respective Fourier spaces. Based on related research, the energy of the time series in the time domain is equal to the energy represented in the frequency domain. This can be expressed by the following mathematical formula:

[0134]

[0135] Where t is the time variable and x(t) represents the signal in the time domain. f is the frequency variable and x(f) is the Fourier transform of x(t) which represents the signal in the frequency domain. Therefore, if most of the energy of a time series is concentrated in a few frequency components, then these components can be used to accurately represent the time series. The Fourier transform helps in extracting trend and periodicity information by decomposing the time series into its frequency components. Therefore, we can downsample the long series and select only the frequency components that are present in the Fourier transform of the short series.

[0136]

[0137] Where x(n) is the time domain signal, X(k) is the frequency domain signal, and N is the total number of samples. From these formulas, we can see that by taking every N values ​​from the long sequence, we can get the corresponding frequency value in the Fourier space of the short sequence.

[0138] y[m]=x[m·N]

[0139] Where x[m·N] is the original long sequence, y[m] is the downsampled short sequence, m is the index in the short sequence, and N is the ratio of the long sequence length to the short sequence length.

[0140] Figure 2 and Figure 3The specific process of global attention is demonstrated. It is worth noting that in order to optimize performance, our invention adopts different methods in training and prediction. In order to better utilize parallel processing capabilities, during training, we did not directly use the Fourier transform of the short sequence as Q, but chose a short sequence matrix randomly sampled from the auxiliary long sequence to calculate the attention weights. In addition, masking the diagonal elements achieves the same effect. We also note that in the original Transformer architecture, the V matrix is ​​used to give specific numerical meanings to lexical components. However, in time series, since the sequence itself has a clear meaning, we omit the V matrix and replace it with the K matrix. Subsequent experiments show that this method improves the accuracy of the model, which can be expressed by the following mathematical formula:

[0141]

[0142] Where Q is the matrix used to calculate the attention weights, representing the query. K serves as both the key and value matrix in this modified version. is the scaling factor of the dot product, d k is the dimension of the key. Therefore, our invention obtains the weight matrix A global , mathematically reflects the similarity of short and long sequences after transformation in Fourier space. This provides a general method for comparing long and short sequences.

[0143] Local Temporal Attention Model

[0144] The local attention in the model refers to capturing the similarity between short sequences and segments of the same length in long sequences. This attention mechanism stems from another assumption: sequences with similar histories are more likely to have similar potential dependencies and thus may have certain future similarities. The difficulty of this problem lies in how to measure this correlation. We are inspired by the calculation of the autocorrelation function (ACF) coefficient.

[0145]

[0146] Where N is the total length of the time series and k is the lag step size. This can be thought of as taking a small portion of the sequence itself as the convolution kernel and convolving it on the original sequence. We can simulate this operation by convolving the auxiliary long sequence using the input short sequence as the convolution kernel.

[0147] However, directly inputting long sequences will result in high time and space complexity, which is not efficient. Since data from the same source exhibits stable fluctuation characteristics, we can consider using Monte Carlo sampling methods to deal with this problem. For a single segment:

[0148]

[0149] Among them, P represents the probability that a single sample is not sampled, N represents the number of segments sampled each time, M represents the total number of segments, and K represents the number of sampling times. Therefore, the expected number of sampling segments can be expressed as:

[0150]

[0151] When K is large enough, it can be considered to be sufficient to reflect the sample characteristics. At the same time, for most time series data that follow a normal distribution:

[0152]

[0153] Among them, z α / 2 is the quantile of the normal distribution, σ is the sample standard deviation, and θ is the mean. N is the sample size, and CI represents the confidence interval. To be precise, local attention first performs a linear transformation on long and short sequences:

[0154] F′=θ(F)

[0155] G′=Φ(G)

[0156] The similarity is calculated through a convolution-like mechanism:

[0157] C=F′*G′

[0158] Among them, F is the matrix obtained by Monte Carlo sampling, G is the short sequence of input, C represents the calculated similarity matrix, Φ and Θ represent linear transformations. Local attention uses a convolution-like calculation method derived from the correlation coefficient to achieve similarity evaluation. By adopting random sampling and parallel computing, it is significantly improved.

[0159] Combining dual attention mechanism for prediction

[0160] First, by matrix multiplication, the global attention and local attention are combined to obtain the weighted result of the final attention, which comprehensively reflects the input sequence x ω The similarity with each auxiliary long sequence in the source domain.

[0161] V=A global ×A local

[0162] Then, the obtained attention matrix V is transformed accordingly through the feedforward neural network:

[0163] FFN(V)=σ(W2σ(W1V+b1)+b2)

[0164] To prevent the gradient from vanishing and improve the training stability and generalization ability of the model, we introduce residual connections in the prediction output. The residual connection adds the original input of the short sequence to the output of the FFN:

[0165]

[0166] Final prediction output It combines weighted information from global and local attention, as well as short sequence features after feedforward neural network processing and residual connection. This enables the model to fully utilize the similarities and dependencies between long and short sequences, thereby achieving accurate prediction of future time series.

[0167] Model implementation and effect verification

[0168] This model is written in Python3, and the neural network structure is built using the interface provided by Pytorch, and runs on Tesla V100 GPU. The present invention has been verified on 7 data sets, including traffic flow data set at traffic intersections, historical energy consumption data set of ten users, and British weather data set. This model is mainly used for the prediction of small sample time series, so it is compared with the six deep learning models currently used for time series prediction, namely PatchTST, LSTNet, N-beats, iTransformer, meta-DKF and DLinear.

[0169] By comparing the prediction accuracy under different sequence lengths, we found that compared with other algorithms, the accuracy of the present invention is improved by about 10%. In addition, the training time of the present invention does not increase significantly with the increase of the prediction sequence length.

[0170] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0171] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A small sample long time series prediction method based on global attention and local attention, characterized in that: include: Step 1: Use the global attention mechanism to input a complete time series, decompose the time series into components across various frequencies through Fourier transform, perform Fourier transform on all auxiliary long sequences to obtain the Fourier transformed matrix, and then perform Fourier transform on the short sequence; downsample the Fourier transformed matrix and align it with the Fourier transformed short sequence; learn weights through the global attention mechanism to obtain the similarity between the short sequence and the auxiliary long sequence in terms of overall fluctuation; Step 2: Adopt the local attention mechanism to randomly select a segment in each auxiliary long sequence to represent the overall features of the auxiliary long sequence, and extract the similarity between the short sequence and different segments in the auxiliary long sequence; Step 3: Combine the global attention mechanism with the local attention mechanism to get the final weight, and obtain the prediction result by combining the feedforward neural network with the residual.

2. The small sample long time series prediction method based on global attention and local attention according to claim 1 is characterized in that: First, input the source domain auxiliary long sequence X = [x1, x2, x3, …, x N ∈ R N×T and the target domain short sequence where t0 << T, N represents the total number of samples; t0 represents the length of the target domain short sequence; T represents the length of the input source domain auxiliary long sequence; R N×T represents the space where the input source domain auxiliary long sequence is located; represents the space where the target domain short sequence is located; Perform fast Fourier transform on the source domain auxiliary long sequence and the target domain short sequence to obtain the corresponding frequency domain representation and Downsample X(f) to x ω (f) The corresponding frequency, get the downsampled matrix X s (f); Then, the weight A is calculated through the global attention mechanism global , and perform Monte Carlo sampling on X to obtain a random sampling sequence X R , using x ω As the convolution kernel, for X R Perform convolution and calculate the local attention weight A local ; Finally, the global and local attention results are combined to obtain the weighted attention weight of the auxiliary long sequence segment: V = A global ×A local , the final prediction output is obtained by superimposing the final attention weight and residual 3. The small sample long time series prediction method based on global attention and local attention according to claim 1 is characterized in that: The step 1 comprises: The distance between the short sequence and the cluster center is measured by the global attention mechanism, and the expression is: D(x ω ,C)=dist(x ω ,C) Among them, x ω =[x1,x2,x3,…,xt] represents a short sequence, C =[c1,c2,c3,…,ct] represents the cluster center mapped to the short sequence space; Then obtain the mapping of the cluster center of the long sequence in the short sequence space, and use Fourier transform to map the long and short sequences to their respective Fourier spaces. The energy of the time series in the time domain is equal to the energy expressed in the frequency domain. The expression is: Where t is the time variable, x(t) represents the signal in the time domain; f is the frequency variable, x(f) is the Fourier transform of x(t), representing the signal in the frequency domain; Downsample the long sequence and select the frequency components that exist in the Fourier transform of the short sequence. The expression is: Among them, x(n) is the time domain signal, X(k) is the frequency domain signal, and N is the total number of samples; Take every α values ​​from the long sequence and get the corresponding frequency value in the Fourier space of the short sequence. The expression is: y[m]=x[m·α] Where x[m·α] is the original long sequence, y[m] is the downsampled short sequence, m is the index in the short sequence, and α is the ratio of the long sequence length to the short sequence length; The attention weight is calculated from the short sequence matrix randomly sampled from the auxiliary long sequence, and the expression is: Among them, Q is the matrix used to calculate the attention weights, representing the query; K is both the key and value matrix; is the scaling factor of the dot product, d k is the dimension of the key; Attention(Q,K) represents the attention weights calculated using the query matrix Q and the key matrix K, which is used to capture the correlation between different positions in the sequence.

4. The small sample long time series prediction method based on global attention and local attention according to claim 1 is characterized in that: The step 2 comprises: The local attention mechanism is used to capture the similarity between short sequences and segments of the same length in long sequences. The expression is: Where N is the total number of samples, k is the lag step size; ρ k represents the autocorrelation coefficient when the sample lag step is k, which measures the similarity of the signal at different lag times; X t represents the signal value at time t; represents the mean value of the signal; For a single fragment: Among them, P represents the probability that a single sample is not sampled, λ represents the number of segments sampled each time, M represents the total number of segments, and C represents the number of sampling times; therefore, the expected number of sampling segments is expressed as: When K is large enough, it is considered to be sufficient to reflect the sample characteristics; at the same time, for most time series data that follow a normal distribution: Among them, z α / 2 is the quantile of the normal distribution, σ is the sample standard deviation, θ is the mean, D is the total number of samples, and CI represents the confidence interval; Local attention performs linear transformation on long and short sequences: F′=θ(F) G′=Φ(G) The similarity is calculated through the convolution mechanism: C=F′*G′ Among them, F is the matrix obtained by Monte Carlo sampling, G is the input short sequence, C represents the calculated similarity matrix, Φ and Θ represent linear transformations, and F′ and G′ are matrices after linear transformation.

5. The small sample long time series prediction method based on global attention and local attention according to claim 1 is characterized in that: The step 3 comprises: First, by means of matrix multiplication, the global attention and local attention are combined to obtain the final weighted result of attention, which comprehensively reflects the input target domain short sequence x ω The similarity with each auxiliary long sequence in the source domain is expressed as: V=A global ×A local Then, the obtained attention matrix V is transformed accordingly through the feedforward neural network, and the expression is: FFN(V)=σ(W2σ(W1V+b1)+b2) Wherein, W1 and W2 represent the weight matrices of the feedforward neural network respectively; b1 and b2 represent the bias vectors of the feedforward neural network respectively; A residual connection is introduced in the prediction output to add the original input of the short sequence to the output of FFN. The expression is:

6. A small sample long time series prediction system based on global attention and local attention, characterized in that: include: Module M1: Using the global attention mechanism, a complete time series is input, and the time series is decomposed into components across various frequencies through Fourier transform. All auxiliary long sequences are Fourier transformed to obtain the Fourier transformed matrix, and then the short sequence is Fourier transformed; the matrix after Fourier transform is downsampled and aligned with the short sequence after Fourier transform; the weights are learned through the global attention mechanism to obtain the similarity between the short sequence and the auxiliary long sequence in terms of overall fluctuation; Module M2: adopts the local attention mechanism to randomly select a segment in each auxiliary long sequence to represent the overall features of the auxiliary long sequence, and extracts the similarity between the short sequence and different segments in the auxiliary long sequence; Module M3: Combine the global attention mechanism with the local attention mechanism to obtain the final weight, and obtain the prediction result by combining the feedforward neural network with the residual.

7. The small sample long time series prediction system based on global attention and local attention according to claim 6, characterized in that: First, input the source domain auxiliary long sequence X = [x1, x2, x3, …, x N ∈ R N×T and the target domain short sequence where t0 << T, N represents the total number of samples; t0 represents the length of the target domain short sequence; T represents the length of the input source domain auxiliary long sequence; R N×T represents the space where the input source domain auxiliary long sequence is located; represents the space where the target domain short sequence is located; Perform fast Fourier transform on the source domain auxiliary long sequence and the target domain short sequence to obtain the corresponding frequency domain representation and Downsample X(f) to x ω (f) The corresponding frequency, get the downsampled matrix X s (f); Then, the weight A is calculated through the global attention mechanism global , and perform Monte Carlo sampling on X to obtain a random sampling sequence X R , using x ω As the convolution kernel, for X R Perform convolution and calculate the local attention weight A local ; Finally, the global and local attention results are combined to obtain the weighted attention weight of the auxiliary long sequence segment: V = A global ×A local , the final prediction output is obtained by superimposing the final attention weight and residual 8. The small sample long time series prediction system based on global attention and local attention according to claim 6, characterized in that: The module M1 comprises: The distance between the short sequence and the cluster center is measured by the global attention mechanism, and the expression is: D(x ω ,C)=dist(x ω ,C) Among them, x ω =[x1,x2,x3,…,xt] represents a short sequence, C =[c1,c2,c3,…,ct] represents the cluster center mapped to the short sequence space; Then obtain the mapping of the cluster center of the long sequence in the short sequence space, and use Fourier transform to map the long and short sequences to their respective Fourier spaces. The energy of the time series in the time domain is equal to the energy expressed in the frequency domain. The expression is: Where t is the time variable, x(t) represents the signal in the time domain; f is the frequency variable, x(f) is the Fourier transform of x(t), representing the signal in the frequency domain; Downsample the long sequence and select the frequency components that exist in the Fourier transform of the short sequence. The expression is: Among them, x(n) is the time domain signal, X(k) is the frequency domain signal, and N is the total number of samples; Take every α values ​​from the long sequence and get the corresponding frequency value in the Fourier space of the short sequence. The expression is: y[m]=x[m·α] Where x[m·α] is the original long sequence, y[m] is the downsampled short sequence, m is the index in the short sequence, and α is the ratio of the long sequence length to the short sequence length; The attention weight is calculated from the short sequence matrix randomly sampled from the auxiliary long sequence, and the expression is: Among them, Q is the matrix used to calculate the attention weights, representing the query; K is both the key and value matrix; is the scaling factor of the dot product, d k is the dimension of the key; Attention(Q,K) represents the attention weights calculated using the query matrix Q and the key matrix K, which is used to capture the correlation between different positions in the sequence.

9. The small sample long time series prediction system based on global attention and local attention according to claim 6, characterized in that: The module M2 comprises: The local attention mechanism is used to capture the similarity between short sequences and segments of the same length in long sequences. The expression is: Where N is the total number of samples, k is the lag step size; ρ k represents the autocorrelation coefficient when the sample lag step is k, which measures the similarity of the signal at different lag times; X t represents the signal value at time t; represents the mean value of the signal; For a single fragment: Among them, P represents the probability that a single sample is not sampled, λ represents the number of segments sampled each time, M represents the total number of segments, and C represents the number of sampling times; therefore, the expected number of sampling segments is expressed as: When K is large enough, it is considered to be sufficient to reflect the sample characteristics; at the same time, for most time series data that follow a normal distribution: Among them, z α / 2 is the quantile of the normal distribution, σ is the sample standard deviation, θ is the mean, D is the total number of samples, and CI represents the confidence interval; Local attention performs linear transformation on long and short sequences: F′=θ(F) G′=Φ(G) The similarity is calculated through the convolution mechanism: C=F′*G′ Among them, F is the matrix obtained by Monte Carlo sampling, G is the input short sequence, C represents the calculated similarity matrix, Φ and Θ represent linear transformations, and F′ and G′ are matrices after linear transformation.

10. The small sample long time series prediction system based on global attention and local attention according to claim 6, characterized in that: The module M3 comprises: First, by means of matrix multiplication, the global attention and local attention are combined to obtain the final weighted result of attention, which comprehensively reflects the input target domain short sequence x ω The similarity with each auxiliary long sequence in the source domain is expressed as: V=A global ×A local Then, the obtained attention matrix V is transformed accordingly through the feedforward neural network, and the expression is: FFN(V)=σ(W2σ(W1V+b1)+b2) Wherein, W1 and W2 represent the weight matrices of the feedforward neural network respectively; b1 and b2 represent the bias vectors of the feedforward neural network respectively; A residual connection is introduced in the prediction output to add the original input of the short sequence to the output of FFN. The expression is:

Images