Long-term time series prediction method based on front-back symmetric decomposition of ESAttention

Abstract

The present invention belongs to the field of deep learning technology, and specifically relates to a long-term time series prediction method based on ESAttention and front-back symmetric decomposition, including the following steps: constructing a neural network model ESSDNet; using two data sets to train and test on the constructed ESSDNet network; establishing evaluation indicators for evaluating the prediction performance of the ESSDNet model. The main framework of the present invention is a convolutional neural network, which uses the idea of ​​front-back symmetric decomposition and feedforward learning to process time series data, and can effectively extract sequential features in time series data; the constructed ESAttention improves the model's ability to capture global features and ensures the correlation between data. The fusion of the two effectively improves the performance of the model's long-term prediction, making the prediction results closer to real data.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a long-term time series prediction method based on front-back symmetric decomposition of ESAttention. Background Art

[0002] A time series is a sequence of data points arranged in chronological order. Comprehensive analysis and mining of this type of data, such as predicting development trends, has been widely used in the real world, such as sensor network monitoring, weather forecasting, economics and finance, disease spread analysis, and power forecasting. In particular, there is an increasing demand for time series prediction for a long period of time in the future. However, with the rapid development of the big data era, the scale of collected data is huge, which brings great challenges to long-term prediction tasks.

[0003] Traditional time series prediction algorithms are mainly based on statistical principles and methods to model and predict time series data. The moving average model (MA) represents the current prediction through a linear combination of random noise and error values ​​of past time series. The autoregressive integrated moving average (ARIMA) model can accurately predict future trends and changes by converting unstable time series into stable time series. However, when faced with complex, nonlinear or high-dimensional time series data, the prediction performance of traditional time series algorithms will drop rapidly and cannot meet the needs of long-term time series prediction tasks.

[0004] In recent years, convolutional neural networks (CNNs), which were first used in the field of computer vision, have been widely used in the field of time series prediction. TCN introduces hole causal convolutions and uses residual connections to allow information to be transferred across layers when passing between networks to solve the problem of information loss caused by convolution when there are too many layers. SCINet adopts a downsampling-convolution-interaction architecture and uses multiple convolution filters to extract unique and valuable temporal features from downsampled subsequences, achieving remarkable results. MICN uses a multi-scale decomposition structure to model and predict different potential patterns respectively. Each pattern uses downsampling convolution and equidistant convolution to extract local features and global correlations, thereby ensuring that the underlying information of time series data is fully utilized. However, methods based on convolutional neural networks still have the following problems: on the one hand, sequential features are one of the most important features of time series data, but existing convolution methods ignore the extraction and learning of sequential features; on the other hand, convolution can capture local and short-term dependencies in the sequence, but may not be able to capture long-term dependencies in the sequence when processing long sequences, and the results achieved in long-term time series prediction tasks are not ideal. Summary of the invention

[0005] In view of the above technical problems, the present invention provides a long-term time series prediction method based on ESAttention with symmetrical decomposition. The idea of ​​symmetrical decomposition of sequences is proposed for the first time to ensure the order of the sequence. At the same time, feedforward learning is proposed so that the second half of the sequence after symmetrical decomposition can learn the information of the first half of the sequence, thereby extracting the sequential features in the sequence; secondly, in order to make up for the unsatisfactory performance of convolution processing long-term time series tasks, the Einstein summation convention (Einsum) is introduced to construct the self-attention mechanism ESAttention based on Einsum, so as to improve the network's ability to extract global features.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] The long-term time series prediction method based on ESAttention's forward and backward symmetric decomposition includes the following steps:

[0008] S1. Construct the ESSDNet, a long-term time series prediction model based on ESAttention and symmetrical decomposition with convolutional neural network as the main feature;

[0009] S2, decompose the sequence using the front-back symmetric decomposition method to ensure that the order of the sequence is not changed;

[0010] S3, using feedforward learning to transfer the information of the first half of the sequence to the second half of the sequence through convolution blocks and Sigmoid function processing for the two subsequences decomposed in S2, so as to extract the sequential features;

[0011] S4. Use the Einstein summation convention Einsum to construct the Einsum-based self-attention mechanism ESAttention to ensure the relevance of data and improve the model's ability to capture global features.

[0012] The main framework of the model in S1 is a convolutional neural network, which includes four parts: trend seasonal term decomposition module, ESAttention module, downsampling module and feedforward learning module. First, the time series data X input into the network model is decomposed into trend seasonal terms. Through decomposition, the time series data X will be decomposed into two parts: trend term X t and the seasonal term X s ; Second, use the linear layer to process X t Get the overall trend characteristics of the time series data; then, X s It is sent to the ESAttention module to extract global features, and then sent to the feedforward learning module in the downsampling module to extract sequential features; finally, the processed X t and X s The final prediction result is obtained by fusion.

[0013] The front-to-back symmetric decomposition method in S2 can decompose the sequence into two subsequences and ensure that the order of the sequence is not changed. The decomposition process is:

[0014] L front ,L back =FBSD(L)

[0015] Where L represents the feature data fed into FFLBlock, FBSD(·) represents the front-to-back symmetric decomposition operation, and L front and L back They respectively represent the first half of the sequence and the second half of the sequence obtained after the forward and backward symmetric decomposition operations.

[0016] The feedforward learning in S3 consists of three convolution blocks, a Sigmoid function and a residual connection; for the decomposed L front , only use different convolutional blocks to process L front The final output is obtained, and the residual connection is used to ensure the performance of the model; and for the decomposed L back , first use the convolution block to process L front Get feature L front_mid , and then use the Sigmoid function to process the feature L front_mid and pass it to L back Get feature L back_mid Then use multiple convolution blocks to process L front_mid Get feature L front_late , and finally the feature L front_late and L back_mid The final output is obtained by fusion, and the residual connection is combined to ensure the learning performance. After the feedforward learning is completed, two new sequences with rich features will be obtained. The feedforward learning process is:

[0017] L front_mid =a(L front )

[0018] L′ front =L front +b(L front_mid )

[0019] L back_mid =L back ⊙Sigmoid(L front_mid )

[0020] L front_late =c(b(L front_mid ))

[0021] L′ back =L back +L back_mid +L front_late

[0022] Where: L′ front Indicates L front The final output after processing by the convolution block, L′ back Indicates L back The final output result after feature fusion, a, b, c represent three different groups of convolution blocks, and ⊙ represents the element product.

[0023] In S4, Einsum can construct an index based on the original position of the data, and then calculate the product of the data based on the index. For complex time series data, using Einsum to construct a self-attention mechanism can more accurately calculate the relationship between the data, while capturing global information and ensuring the correlation between the data. The calculation process of ESAttention is:

[0024]

[0025] Where: Q, K, V represent the query, key, and value of each data respectively, and d represents the feature dimension of the input data.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention uses the idea of ​​front-to-back symmetric decomposition and feedforward learning to effectively extract sequential features from time series data, and uses ESAttention to improve the model's ability to capture global features and the correlation between data, making the predicted results more consistent with real data. The present invention obtains good quantitative results on the two evaluation indicators of MSE and MAE in the transformer data set, and also shows an intuitive feeling that is closer to the real value in the qualitative results. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0029] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0030] Figure 1 The overall architecture diagram of ESSDNet of the present invention;

[0031] Figure 2 It is a structural diagram of ESAttention in the ESSDNet model of the present invention;

[0032] Figure 3 It is a structural diagram of the three convolution blocks a, b, and c in the ESSDNet model of the present invention;

[0033] Figure 4 This is a qualitative comparison result diagram of the ESSDNet model used in the present invention and other time series prediction methods on the transformer data set. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0035] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0036] The present invention is implemented under the pytorch deep learning framework. The present invention provides an ESSDNet long-term time series prediction method, which specifically includes the following steps:

[0037] 1. Data preparation

[0038] The data samples of the present invention include two time series data sets:

[0039] Dataset I: The ETTh dataset provided by the State Grid Corporation of China, including two datasets, ETTh1 and ETTh2. ETTh1 and ETTh2 are data collected every hour from transformers in different counties of the same province in China from July 2016 to July 2018. Each dataset has 17,420 data, with a total of 8 dimensions, 1 time dimension and 7 feature dimensions. Dataset II: The ETTm dataset provided by the State Grid Corporation of China, including two datasets, ETTm1 and ETTm2. ETTm1 and ETTm2 are data collected every 15 minutes from transformers in different counties of the same province in China from July 2016 to July 2018. Each dataset has 69,680 data, with a total of 8 dimensions, 1 time dimension and 7 feature dimensions.

[0040] The above data set is standardized. The mean and variance of the original data set are first calculated, and then the original data is subtracted from the mean and divided by the variance to obtain the normalized data, which ensures rapid convergence during training and improves the prediction performance of the network.

[0041] 2. Model construction

[0042] The main framework of the constructed ESSDNet network model is a convolutional neural network structure. The specific network model structure is as follows: Figure 1 As shown in the figure, the ESSDNet network model contains two branches: trend term and season term. The trend term branch contains 1 Revin normalization module, 1 linear layer module and an anti-Revin normalization module. The season term branch contains 1 Revin normalization module, 1 ESAttention module, 1 downsampling module FSDBlock, 1 point-by-point convolution module with a convolution kernel of 1 and a step size of 1, and an anti-Revin normalization module. The ESAttention module is shown in the figure. Figure 2 As shown in Figure 2, the downsampling module contains multiple feedforward learning modules FFLBlock and 1 ESAttention module. The feedforward learning module FFLBlock contains 3 convolution blocks, 1 Sigmoid function and 2 residual connections. The convolution block is as follows: Figure 3As shown in the figure, the convolution block contains 1 tensor filling module, 1 1D convolution with a convolution kernel of 5 and a stride of 1, 1 LeakyReLu activation layer, 1 Dropout layer, 1 1D convolution with a convolution kernel of 3 and a stride of 1, and a Tanh layer. When the time series data with a length of 512 is input into the network, it will first be decomposed into trend items and seasonal items by the trend season item decomposition module. Then, for the trend item, only a simple linear layer is used for processing. For the seasonal item, it is first sent to the ESAttention module to extract global features, and then sent to the feedforward learning module in the downsampling module to extract sequential features, and then output through the point-by-point convolution module; finally, the processed trend item and seasonal item results are fused to obtain the final prediction result.

[0043] The feedforward learning process is as follows:

[0044] L front_mid =a(L front )

[0045] L′ front =L front +b(L front_mid )

[0046] L back_mid =L back ⊙Sigmoid(L front_mid )

[0047] L front_late =c(b(L front_mid ))

[0048] L′ back =L back +L back_mid +L front_late

[0049] Where: L′ front Indicates L front The final output after processing by the convolution block, L′ back Indicates L back The final output result after processing by the convolution block, a, b, c represent three different groups of convolution blocks, and ⊙ represents the element-wise product.

[0050] The calculation process of ESAttention is as follows:

[0051]

[0052] Where: Q, K, V represent the query, key, and value of each data respectively, and b represents the feature dimension of the input data.

[0053] 3. Model training

[0054] In the constructed ESSDNet network model, L1 loss is used as the loss function of the ESSDNet network model. This loss is not easily affected by outliers and has good robustness. The L1 loss calculation formula of the model is:

[0055]

[0056] Among them, N represents the amount of data in the data set, y i represents the true value, y′ i Represents the predicted value.

[0057] 4. Test results

[0058] The method for testing the long-term time series prediction results is as follows: for the ETTh and ETTm data sets, the data is first standardized, and then the standardized data is passed through the ESSDNet model to obtain the predicted data. Finally, the corresponding evaluation indicators are used to evaluate the prediction performance of the predicted data and the real data.

[0059] 5. Model Evaluation

[0060] The mean square error (MSE) and mean absolute error (MAE) are used to evaluate the performance of the model. The lower the values ​​of the two indicators, the better the prediction performance.

[0061] Table 1 Comparison results of different methods on the ETTh1 dataset

[0062]

[0063] Table 2 Comparison results of different methods on the ETTh2 dataset

[0064]

[0065] Table 3 Comparison results of different methods on the ETTm1 dataset

[0066]

[0067]

[0068] Table 4 Comparison results of different methods on the ETTm2 dataset

[0069]

[0070] The long-term prediction results on the ETTh and ETTm datasets are shown in Tables 1, 2, 3, 4 and Figure 4As shown in the table, 96, 192, 336, and 720 represent four different prediction step sizes, and the optimal index is in bold. From the table and figure, it can be found that the proposed ESSDNet method is basically superior to other methods in terms of both evaluation indicators and prediction effects.

[0071] For the Informer model, the model introduces sparse bias in the attention module, which improves the computational efficiency of the model, but ignores important features in the sequence, resulting in poor performance; for the SCINet model, the model uses odd-even decomposition and interactive learning methods to capture features, which causes the loss of sequence order features and leads to performance degradation; for the Autoformer model, the model uses an associative attention mechanism to capture sequence information, but the features extracted by the associative mechanism are insufficient, resulting in no significant improvement in model performance; for the Fedformer model, the model is based on frequency domain decomposition, which makes the model overly dependent on the frequency domain information of the data, thereby ignoring the time domain information, resulting in limited performance of the model; for the MICN model, the model uses convolution to extract local features and global features respectively, but the ability of convolution to capture global features is still limited, so the model still does not perform well in long-term prediction performance.

[0072] For the PatchTST model, when the prediction length is 96 and 192, some indicators of the ESSDNet model are higher than those of the PatchTST model. This is because the PatchTST model divides the sequence into multiple patches to characterize the local features of the sequence, which greatly improves the local correlation between the data and thus achieves better performance. However, when the prediction length is 336 and 720, the performance of the ESSDNet model is basically better than that of the PatchTST model. This is because the feedforward learning proposed by the ESSDNet model can well extract the sequential features of the sequence, and the constructed ESAttention improves the limitation of the convolutional neural network in extracting insufficient global information. The two together establish a long-term dependency relationship between sequences. According to the overall experimental results, the ESSDNet method has certain advantages over other methods in long-term time series prediction tasks.

[0073] Only the preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the protection scope of the present invention.

Claims

1. The long-term time series prediction method based on ESAttention and front-back symmetric decomposition is characterized by: The following steps are involved: S1. Construct the ESSDNet, a long-term time series prediction model based on ESAttention and symmetrical decomposition with convolutional neural network as the main feature; S2. Decompose the sequence using a front-to-back symmetric decomposition method to ensure that the order of the sequence is not changed; the front-to-back symmetric decomposition method in S2 can decompose the sequence into two subsequences and ensure that the order of the sequence is not changed. The decomposition process is: L front ,L back =FBSD(L) Where L represents the feature data fed into FFLBlock, FBSD(·) represents the front-to-back symmetric decomposition operation, and L front and L back Respectively represent the first half of the sequence and the second half of the sequence obtained after the front and back symmetric decomposition operations; S3, using feedforward learning to transfer the information of the first half of the sequence to the second half of the sequence through convolution blocks and Sigmoid function processing for the two subsequences decomposed in S2, so as to extract the sequential features; S4. Use the Einstein summation convention Einsum to construct the Einsum-based self-attention mechanism ESAttention to ensure the relevance of data and improve the model's ability to capture global features.

2. The method for long-term time series prediction based on ESAttention and front-back symmetric decomposition according to claim 1 is characterized in that: The main framework of the model in S1 is a convolutional neural network, which includes four parts: trend seasonal term decomposition module, ESAttention module, downsampling module and feedforward learning module. First, the time series data X input into the network model is decomposed into trend seasonal terms. Through decomposition, the time series data X will be decomposed into two parts: trend term X t and the seasonal term X s ; Second, use the linear layer to process X t Get the overall trend characteristics of time series data; Then, X s It is sent to the ESAttention module to extract global features, and then sent to the feedforward learning module in the downsampling module to extract sequential features; finally, the processed X t and X s The final prediction result is obtained by fusion.

3. The method for long-term time series prediction based on ESAttention with forward and backward symmetric decomposition according to claim 1, characterized in that: The feedforward learning in S3 consists of three convolution blocks, a Sigmoid function and a residual connection; for the decomposed L front , only use different convolutional blocks to process L front The final output is obtained, and the residual connection is used to ensure the performance of the model; and for the decomposed L back , first use the convolution block to process L front Get feature L front_mid , and then use the Sigmoid function to process the feature L front_mid and pass it to L back Get feature L back_mid Then use multiple convolution blocks to process L front_mid Get feature L front_late , and finally the feature L front_late and L back_mid The final output is obtained by fusion, and the residual connection is combined to ensure the learning performance. After the feedforward learning is completed, two new sequences with rich features will be obtained. The feedforward learning process is: L front_mid =a(L front ) The front =L front +b(L front_mid ) L back_mid =L back ⊙Sigmoid(L front_mid ) L front_late =c(b(L front_mid )) L' back =L back +L back_mid +L front_late Where: L′ front Indicates L front The final output after processing by the convolution block, L′ back Indicates L back The final output result after feature fusion, a, b, c represent three different groups of convolution blocks, and ⊙ represents the element product.

4. The method for long-term time series prediction based on ESAttention and front-back symmetric decomposition according to claim 1 is characterized in that: In S4, Einsum can construct an index based on the original position of the data, and then calculate the product of the data based on the index. For complex time series data, using Einsum to construct a self-attention mechanism can more accurately calculate the relationship between the data, while capturing global information and ensuring the correlation between the data. The calculation process of ESAttention is: Where: Q, K, V represent the query, key, and value of each data respectively, and d represents the feature dimension of the input data.

Images