Time series prediction method of multi-time scale channel mixed attention mechanism

Abstract

The invention discloses a time series prediction method of a multi-time scale channel mixed attention mechanism, and relates to the technical field of time series prediction and deep learning. Specifically, weighted processing is carried out on extracted features of time sequence data through a channel mixed attention mechanism. The mechanism can dynamically adjust the weight of the feature according to the importance of the feature, and ensures that the key feature plays a greater role in prediction. Inputting the time sequence data into a multi-time scale convolutional neural network to obtain feature vectors under different time scales; and fusing the weighted features to obtain comprehensive multi-time scale features, and completing the training process of the time sequence prediction model. According to the method, the accuracy and adaptability of time sequence prediction are improved, and complex modes and trends in time sequence data can be captured more accurately. Through the method, the method can play a role in various application scenes needing high-precision prediction.

Description

Technical Field

[0001] The present invention relates to the fields of image data processing, time series prediction and deep learning technology, and in particular to a Transformer-based time series prediction method. Background Art

[0002] In the prediction of image data time series, the requirements for the extraction and use of time information are constantly increasing. In order to address these challenges, researchers are constantly exploring new methods to better capture the complex patterns and time dependencies in time series data. Existing emerging methods not only focus on a single time scale or feature, but combine a variety of technical means to strive to comprehensively and accurately extract key information from the data. For example, in the process of processing time series data, researchers have begun to introduce advanced technologies such as multi-time scale analysis, attention mechanisms, and multi-layer neural networks. These technologies can more deeply explore the potential relationships and features in the data through multi-angle and multi-level analysis, especially when faced with time series data with highly nonlinear and diverse patterns, showing strong adaptability and flexibility.

[0003] In addition, with the advancement of computing power and data processing technology, researchers are also exploring how to more effectively process large-scale data sets and real-time data, thereby improving the efficiency and scalability of models while maintaining prediction accuracy. These efforts have not only promoted the advancement of time series prediction technology, but also brought new possibilities to multiple application fields such as finance, medicine, and transportation. For example, in the financial field, researchers can use these technologies to more accurately predict market trends and provide more reliable support for investment decisions; in the medical field, doctors can develop more effective treatment plans through in-depth analysis of patients' historical data; in the transportation field, real-time traffic forecasts can help managers better plan and optimize traffic flow and reduce congestion and delays.

[0004] Traditional time series forecasting methods, such as autoregressive models (AR), moving average models (MA), and autoregressive integrated moving average models (ARIMA), perform well when processing linear time series data, but are unable to cope with complex nonlinear and multi-scale time dependencies. With the development of deep learning technology, convolutional neural networks (CNN) and recurrent neural networks (RNN) have made significant progress in time series forecasting. CNN can effectively extract local features, while RNN, especially long short-term memory networks (LSTM) and gated recurrent units (GRU), excel in capturing long-term dependencies. However, a single feature extraction method still cannot fully capture the multi-scale features and long-term and short-term dependencies in time series data.

[0005] Through these innovations and developments, time series forecasting technology is constantly breaking through the limitations of traditional methods and moving towards a more intelligent and efficient direction. In the future, with the introduction and application of more new technologies, time series forecasting is expected to play an important role in more fields, further improve the accuracy and reliability of forecasts, and provide greater convenience and support for people's production and life. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a time series prediction method with a multi-time-scale channel mixed attention mechanism, which solves the problem that the prior art has defects in target training and sample selection, resulting in low accuracy of prediction results.

[0007] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a time series prediction method of a multi-time scale channel hybrid attention mechanism, comprising the following steps:

[0008] S1, perform instance normalization on the input time series data and divide it into blocks;

[0009] S2, project the divided time series data and add position embedding information;

[0010] S3, input the projected and position-embedded time series data into the channel attention module, extract features through maximum pooling and average pooling, and process them through a shared multi-layer perceptron MLP to obtain the channel attention weight;

[0011] S4, input the data processed by the channel attention module into the spatial attention module, extract the spatial features through the convolution layer and the pooling layer including the maximum pooling and the average pooling in one step, and calculate the spatial attention weight;

[0012] S5, input the data processed by the spatial attention module into the multi-time scale hybrid module, which includes a multi-time scale hybrid attention mechanism, and processes it through addition and normalization layers and feed-forward network components to extract multi-time scale features;

[0013] S6. Flatten the multi-time scale features output by the multi-time scale mixing module and further process them through a linear layer to generate the final prediction result.

[0014] The beneficial effects of the above scheme are as follows: the present invention uses a channel hybrid attention mechanism to perform weighted processing on the extracted features of the time series data. The mechanism can dynamically adjust the weight of the features according to their importance, ensuring that key features play a greater role in prediction, and input the time series data into a multi-time scale convolutional neural network to obtain feature vectors at different time scales, and fuse the weighted features to obtain comprehensive multi-time scale features. Finally, these multi-time scale features are input into a deep learning model for training to complete the training process of the time series prediction model. The present invention solves the problem that the prior art has defects in target training and sample selection, resulting in low accuracy of prediction results.

[0015] Furthermore, the normalization processing and block division in S1 include:

[0016] S11, normalize the time series data to the range of 0,1;

[0017] S12, dividing the normalized time series data into segments;

[0018] The time series data segmentation into segments in S12 further includes:

[0019] S121, dividing the long time series into multiple subsequences of fixed length;

[0020] S122, selecting a block length P and a non-overlapping area S between blocks, wherein the block length determines the number of time steps of each subsequence;

[0021] S123, the number of blocks N is:

[0022]

[0023] Among them, P is the length of each block, that is, the number of time steps contained in each block, and the non-overlapping area S between blocks. If the time series length L cannot divide the block length P, it can be padded at the end, usually by repeating the last value until the length meets the block requirements.

[0024] The beneficial effects of the above further scheme are: by preprocessing the time series data, the model performance and generalization ability are improved, the data processing is simplified, the computational complexity is reduced, and the visualization and interpretation of the data are facilitated.

[0025] Furthermore, adding the position embedding information in S2 includes:

[0026] S21, generating a position embedding vector using sine function and cosine function;

[0027] S22, adding the generated position embedding vector and the input time series data;

[0028] S23, generate a unique embedding vector for each position in the time series data;

[0029] The beneficial effect of the above further scheme is that these position embeddings help the model understand the temporal dependencies of the data, allowing it to better capture patterns and trends in the sequence. This information can also enhance the generalization ability of the model, allowing it to perform better on data from different time periods and improve the accuracy of predictions.

[0030] Furthermore, the channel attention module used in S3 further includes:

[0031] S31, using global average pooling to extract macro information;

[0032] S32, global maximum pooling captures key clues of features and helps infer more refined channel attention;

[0033] S33, combining the two, global average pooling obtains macro information, and global maximum pooling obtains local information;

[0034] The beneficial effect of the above further scheme is: by adaptively adjusting the feature channel weights to enhance the model's feature representation ability and generalization performance, improve the accuracy of the prediction task, while reducing computing resource consumption and enhancing the interpretability of the model's decision process.

[0035] Furthermore, the use of the spatial attention module in S4 further includes:

[0036] S41, extracting two input feature maps of the same dimension size through channel-based global maximum pooling and global average pooling;

[0037] S42, concatenate the two feature maps in the channel dimension, integrate the information of different channels, and use convolution to reduce the dimension;

[0038] S43, generate spatial attention features through sigmoid function activation and multiply it with the input feature map;

[0039] The beneficial effect of the above further scheme is: convolution operation is performed on the refined features to extract spatial features. The convolution layer can capture the local spatial pattern of the data and help extract spatial dependencies.

[0040] Furthermore, the use of a multi-time scale mixing module in S5 further includes:

[0041] S51, representing the input time series data as subsequences across different time scales, performing convolution on the subsequences of each time scale, where the convolution kernel is a unimodal sequence with the same size as the subsequence;

[0042] S52, input the convolutional features at each time scale into the multi-head self-attention mechanism, each attention head focuses on a different time range;

[0043] The beneficial effects of the above further scheme are: significantly improving the model's feature representation ability, temporal dependency capture ability, generalization ability and robustness, thereby improving the overall performance.

[0044] Furthermore, the sine function PE(t,2i) and the cosine function PE(t,2i+1) are used to generate the embedding vector:

[0045]

[0046]

[0047] Here, t is the position in the sequence, i is the dimension index, and D is the dimension of the embedding vector.

[0048] The beneficial effect of the above further scheme is: the position embedding vector generated by the sine and cosine functions retains the order information of the time series by encoding unique position information for each time step, ensuring that the model can understand the timing characteristics of the sequence data. This method utilizes the periodic characteristics of the sine and cosine functions and can effectively capture periodic changes and patterns in the data.

[0049] Furthermore, the attention weight vector ω in the channel attention module is:

[0050] ω=Sigmoid(FC2(ReLU(FC1(z))))

[0051] Among them, the input time series data is X, and its size is C×T, where C represents the number of channels and T represents the number of time steps. A convolutional neural network (CNN) is used to extract features from the input data to generate a feature map F, whose size is C×H×W, where H represents height and W represents width. A global average pooling operation is applied to the feature map F to obtain the average eigenvalue z of each channel, whose size is C. The average eigenvalue z is input into two fully connected layers (FC) for feature transformation and activation to obtain the channel attention weight vector ω.

[0052] The beneficial effects of the above further scheme are: significantly enhancing feature extraction capabilities, highlighting important features, improving model performance and accuracy, enhancing the generalization ability of the model, and reducing computing resource consumption. The convolutional neural network effectively extracts complex patterns and local correlations from the input time series data. The global average pooling operation reduces the data dimension and retains global information. The channel attention mechanism adaptively adjusts the weights of each channel and optimizes the feature representation, so that the model can better handle complex patterns and local correlations in time series data.

[0053] Furthermore, the spatial attention feature map A in the spatial attention module is:

[0054] A=Sigmoid(Conv(ReLU(F merge )))

[0055] Among them, the feature map F merge Two different feature maps F are obtained by maximum pooling and average pooling respectively. max and F avg Combined.

[0056] The beneficial effects of the above further scheme are: significantly improving the spatial feature extraction capability and overall performance of the model, the convolution operation captures the local spatial pattern, the maximum pooling and average pooling extract the significant features and global information respectively, and the generated spatial attention weights are applied to the channel refined features by element-by-element multiplication, so that the model can dynamically adjust the weight of each position and optimize the feature representation.

[0057] Furthermore, each subsequence at each time scale is convolved, where the convolution kernel is a unimodal sequence K(t) of the same size as the subsequence:

[0058]

[0059] Among them, K(t) is the convolution kernel value at time t, t is the current time step, μ is the center point of the time distribution, in the context of the convolution kernel, it is usually the center position of the convolution kernel, and σ is the width or expansion of the distribution.

[0060] The beneficial effects of this scheme are: capturing long-term and short-term dependencies, enhancing feature extraction capabilities, improving the robustness and accuracy of the model, optimizing the generalization ability of the model, and reducing information loss, integrating information at different time scales, making the model more efficient and accurate when processing complex time series data, thereby comprehensively improving the overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a time series prediction method with a multi-time scale channel mixed attention mechanism;

[0062] Figure 2 Schematic diagram of the channel attention module;

[0063] Figure 3 Schematic diagram of the spatial attention module;

[0064] Figure 4 Schematic diagram of the multi-time scale mixing module. DETAILED DESCRIPTION

[0065] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0066] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Those skilled in the art should know that the following specific embodiments or specific implementations are a series of optimized settings listed by the present invention to further explain the specific content of the invention, and these settings can be combined or used in association with each other, unless the present invention clearly states that some or a specific embodiment or implementation cannot be associated or used together with other embodiments or implementations. At the same time, the following specific embodiments or implementations are only used as the most optimized settings, and are not to be understood as limiting the scope of protection of the present invention.

[0068] like Figure 1 As shown, a time series prediction method of a multi-time scale channel mixed attention mechanism includes the following steps:

[0069] S1: Perform instance normalization on the input time series data and perform block division to divide the long time series into multiple subsequences of fixed length to simplify subsequent processing and modeling. The block division process includes selecting the block length P and the non-overlapping area S between blocks, where the block length determines the number of time steps of each subsequence. Appropriate block division can not only effectively reduce the dimension of the data and highlight local features, but also increase the diversity of samples through overlapping blocks, thereby improving the robustness and generalization ability of the model;

[0070] S2: Linearly project the divided time series data to better utilize the sine function PE(t,2i) and the cosine function PE(t,2i+1) to generate a position embedding vector, and combine it with the input feature. This method generates a unique embedding vector for each position in the sequence, so that the temporal relationship in the sequence can be retained and identified during model processing to maintain the temporal information of the sequence;

[0071] S3: The projected and position-embedded data is input into the channel attention module. Global average pooling is used to extract macro information, while global maximum pooling can capture key clues of features and help infer more refined channel attention. By combining the two, AvgPool obtains macro information and MaxPool obtains local information. At the same time, MaxPool can also improve the robustness of the network, highlight important features, and improve the recognition ability of the model.

[0072] S4: The data processed by the channel attention module is input into the spatial attention module. Two input feature maps of the same size are extracted through channel-based global maximum pooling and global average pooling, and they are spliced ​​in the channel dimension to integrate the information of different channels. Convolution is used for dimensionality reduction to capture a wider range of spatial features. The spatial attention feature is generated through sigmoid function activation and multiplied with the input feature map to highlight important spatial information areas, thereby improving the spatial feature expression ability of the model;

[0073] S5: The data processed by the spatial attention module is input into the multi-time scale hybrid module, the input time series data is represented as subsequences across different time scales, and the subsequences of each time scale are convolved. The convolution kernel is a unimodal sequence with the same size as the subsequence. The convolution features at each time scale are input into the multi-head self-attention mechanism. Each attention head focuses on a different time range, so that this variant of the self-attention mechanism can capture information in the time series data at multiple time scales.

[0074] S6: Flatten the features output by the multi-time scale mixing module, flatten the height, width and number of channels of the output features into a long vector, so as to facilitate subsequent linear layer processing, and then further extract and process features through one or more linear layers to finally generate the prediction results of the model.

[0075] like Figure 1 As shown, the number of blocks N mentioned in S1 is:

[0076]

[0077] Among them, P is the length of each block, that is, the number of time steps contained in each block, and the non-overlapping area S between blocks. If the time series length L cannot divide the block length P, it can be padded at the end, usually by repeating the last value until the length meets the block requirements.

[0078] like Figure 2 As shown, the attention weight vector ω in the channel attention module is:

[0079] ω=Sigmoid(FC2(ReLU(FC1(z)))) (2)

[0080] The input time series data is X, and its size is C×T, where C represents the number of channels and T represents the number of time steps. First, a convolutional neural network (CNN) is used to extract features from the input data to generate a feature map F, whose size is C×H×W, where H represents the height and W represents the width. Subsequently, a global average pooling operation is applied to the feature map F to obtain the average feature value z of each channel, whose size is C. Then, the average feature value z is input into two fully connected layers (FC) for feature transformation and activation to obtain the channel attention weight vector ω.

[0081] like Figure 3 As shown, the spatial attention feature map A in the spatial attention module is:

[0082] A=Sigmoid(Conv(ReLU(F merge ))) (3)

[0083] Among them, the feature map F merge Two different feature maps F are obtained by maximum pooling and average pooling respectively. max and F avg Combined.

[0084] like Figure 4 As shown, each subsequence at each time scale is convolved, where the convolution kernel is a unimodal sequence with the same size as the subsequence:

[0085]

[0086] Where K(t) is the convolution kernel value at time t, t is the current time step, and μ is the center point of the time distribution. In the context of the convolution kernel, it is usually the center position of the convolution kernel. σ is the width or spread of the distribution. The input time series data is divided into different time scales, and each time scale corresponds to a subsequence. In this paper, three time scales are considered: the input time series is x1, x2, ..., x t ,...,x n , the current time step is t, then the information of the current time step is x t When the time scale is 2, extract x t-2 ,x t-1 ,x t ,x t+1 ,x t+2 , which includes the data of the five time steps before and after the current time step. When the time scale is 4, extract x t-4 ,x t-3 ,x t-2 ,x t-1 ,x t ,x t+1 ,x t+2 ,xt+3 ,x t+4 , that is, the data of the four time steps before and after the current time step, a total of nine time steps. Use a convolution kernel with the same length as the subsequence. The convolution kernel is initialized as a unimodal sequence, and a one-dimensional convolution operation is performed on each subsequence. The convolution kernels are w1, w2, ..., w k , the subsequence is s1,s2,...,s k , then the result of the convolution operation is The convolutional features at each time scale are connected to generate a feature representation containing multi-time scale information. The output of the multi-time scale mixed attention module is added to the original input to generate the residual connection feature.

[0087] The present invention proposes a time series prediction method with a multi-time scale channel hybrid attention mechanism, which captures long-term and short-term dependencies through a multi-time scale design, uses channel hybrid technology to interact and fuse information between different feature channels, and dynamically allocates weights in combination with the attention mechanism to enhance feature expression and capture key information. The specific implementation includes using convolutional neural networks and Transformer networks to extract time series features, performing feature processing at different time scales, and weighting features through a channel hybrid attention module, and finally outputting prediction results through a decoder. This method has technical advantages such as capturing multi-scale dependencies, enhancing feature fusion, and dynamic weight allocation. It is suitable for various time series prediction tasks such as financial market forecasting, weather forecasting, and industrial process monitoring, and has wide applicability and high flexibility. The protection scope of the present invention is not limited to specific embodiments, but covers all variations and combinations that do not deviate from the essence of the present invention.

[0088] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the invention.

Claims

1. A time series prediction method with a multi-time scale channel mixed attention mechanism, characterized in that: The following steps are involved: S1. Perform instance normalization on the input time series data and divide it into blocks for subsequent processing; S2, project the divided time series data and add position embedding information; S3, input the projected and position-embedded time series data into the channel attention module, extract features through maximum pooling and average pooling, and process them through a shared multi-layer perceptron to obtain channel attention weights; S4, input the data processed by the channel attention module into the spatial attention module, extract the spatial features through the convolution layer and the pooling layer including the maximum pooling and the average pooling in one step, and calculate the spatial attention weight; S5, input the data processed by the spatial attention module into the multi-time scale hybrid module, which includes a multi-time scale hybrid attention mechanism, and processes it through addition and normalization layers and feed-forward network components to extract multi-time scale features; S6, flattening the multi-time scale features output by the multi-time scale mixing module, and further processing them through a linear layer to generate the final prediction result; The normalization processing and block division in S1 further includes: S11, normalize the time series data to the range of 0,1; S12, dividing the normalized time series data into segments; The time series data segmentation into segments in S12 further includes: S121, dividing the long time series into multiple subsequences of fixed length; S122, selecting a block length P and a non-overlapping area S between blocks, wherein the block length determines the number of time steps of each subsequence; S123, the number of blocks N is: Among them, P is the block length of each block, S represents the non-overlapping area between blocks, and L represents the length of the time series; if L cannot divide the block length P, padding is performed at the end until the length meets the block requirements.

2. The time series prediction method of multi-time scale channel mixed attention mechanism according to claim 1 is characterized in that: The adding of location embedding information in S2 further comprises: S21, generating a position embedding vector using sine function and cosine function; S22, adding the generated position embedding vector and the input time series data; S23. Generate a unique embedding vector for each position in the time series data.

3. The time series prediction method of multi-time scale channel mixed attention mechanism according to claim 1 is characterized in that: The channel attention module used in S3 further includes: S31, using global average pooling to extract macro information; S32, global maximum pooling captures key clues of features and helps infer more refined channel attention; S33. Combining the two, global average pooling obtains macro information, and global maximum pooling obtains local information.

4. The time series prediction method of multi-time scale channel mixed attention mechanism according to claim 1 is characterized in that: The spatial attention module used in S4 further includes: S41, extracting two input feature maps of the same dimension size through channel-based global maximum pooling and global average pooling; S42, concatenate the two feature maps in the channel dimension, integrate the information of different channels, and use convolution to reduce the dimension; S43. Generate spatial attention features through sigmoid function activation and multiply them with the input feature map.

5. The time series prediction method of multi-time scale channel mixed attention mechanism according to claim 1 is characterized in that: The multi-time scale mixing module used in S5 further comprises: S51, representing the input time series data as subsequences across different time scales, performing convolution on the subsequences of each time scale, where the convolution kernel is a unimodal sequence with the same size as the subsequence; S52. The convolutional features at each time scale are input into the multi-head self-attention mechanism, where each attention head focuses on a different time range.

6. The time series prediction method of multi-time scale channel mixed attention mechanism according to claim 2 is characterized in that: The S21 further includes: the sine function is: The cosine function is: Here, t is the position in the sequence, i is the dimension index, and D is the dimension of the embedding vector.

7. The time series prediction method of multi-time scale channel mixed attention mechanism according to claim 3 is characterized in that: The attention weight vector ω in the channel attention module is: ω=Sigmoid(FC2(ReLU(FC1(z)))) Among them, the input time series data is X, and its size is C×T, where C represents the number of channels and T represents the number of time steps. First, a convolutional neural network is used to extract features from the input data to generate a feature map F with a size of C×H×W, where H represents height and W represents width. Subsequently, a global average pooling operation is applied to the feature map F to obtain the average feature value z of each channel with a size of C. Then, the average feature value z is input into two fully connected layers for feature transformation and activation to obtain the channel attention weight vector ω.

8. The time series prediction method of multi-time scale channel mixed attention mechanism according to claim 4 is characterized in that: The input feature map F of each channel is multiplied element-by-element with its corresponding attention weight w to obtain the weighted feature map F′. The spatial attention feature map A in the spatial attention module is: A=Sigmoid(Conv(ReLU(F merge ))) Among them, the feature map F merge Two different feature maps F are obtained by maximum pooling and average pooling respectively. max and F avg Combined.

9. The time series prediction method of multi-time scale channel mixed attention mechanism according to claim 5 is characterized in that: The multi-timescale mixture module enables the model to focus on various scales in different time ranges, thereby more comprehensively capturing long-term and short-term dependencies in sequence data. Subsequently, each subsequence at each time scale is convolved, where the convolution kernel is a unimodal sequence of the same size as the subsequence: Among them, K(t) is the convolution kernel value at time t, t is the current time step, μ is the center point of the time distribution, in the context of the convolution kernel, it is usually the center position of the convolution kernel, and σ is the width or expansion of the distribution.

Images