Feature pyramid time sequence prediction method based on cross attention

Abstract

The invention discloses a feature pyramid time sequence prediction method based on cross attention. The method comprises the steps of performing normalization preprocessing on time sequence data; carrying out feature extraction on the time series data by using a convolutional neural network, and carrying out global average pooling operation after each convolution; performing multi-stage feature extraction on the extracted features, extracting trend features by using average pooling, taking the output of the previous step as the input of the next average pooling, letting x1 = x, performing average pooling operation on x1 to obtain x2, repeating the operation until x4 is obtained, and performing processing construction by using a feature pyramid module to obtain a multi-scale time sequence feature expression X; a plurality of full-connection layer prediction target sequences Y = [y1, y2, y3, y4] are introduced, the output yi of the current layer and the output yi-1 of the previous layer are spliced, and the full-connection layer is used for processing so that the output dimension of the full-connection layer can be matched with yi; and carrying out prediction by using a Transform based on cross attention. The method can effectively capture the long-distance dependency relationship of data, and improves the prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the field of machine learning and artificial intelligence technology, and in particular to a feature pyramid time series prediction method based on cross attention. Background Art

[0002] With the development of deep learning technology, especially the emergence of RNN, LSTM and GRU, the accuracy of time series forecasting has been significantly improved, which can capture long-term dependencies in the data and perform well in processing nonlinear time series data.

[0003] Although deep learning has made progress in time series forecasting, existing technologies still have limitations: the lack of an effective multi-scale feature fusion mechanism leads to poor performance of the model when processing data at different time scales; existing models still face challenges in capturing long-distance dependencies in time series data; and it is difficult to balance the prediction efficiency and model complexity of existing methods, especially when dealing with large-scale time series data. Summary of the invention

[0004] The purpose of the present invention is to provide a feature pyramid time series prediction method based on cross-attention, which can effectively capture the long-distance dependencies of data and greatly improve the accuracy of prediction.

[0005] In order to achieve the above object, the present invention provides a feature pyramid time series prediction method based on cross attention, the method comprising:

[0006] Normalize and preprocess the time series data;

[0007] Use convolutional neural networks to extract features from time series data and perform a global average pooling operation after each convolution;

[0008] The extracted features are extracted using multi-level features, and the trend features are refined using average pooling. The output of the previous step is used as the input of the next average pooling. Let x1 = x, and perform an average pooling operation on x1 to obtain x2. Repeat the above operation until x4 is obtained, and use the feature pyramid module to construct a multi-scale temporal feature representation X;

[0009] Introduce multiple fully connected layers to predict the target sequence Y = [y1, y2, y3, y4], and convert the output y of the current layer i With the output y of the previous layer i-1 concatenate, and use the fully connected layer to process so that the output dimension of the fully connected layer is the same as y i Match;

[0010] Use the criss-cross attention based Transformer for prediction.

[0011] Preferably, the normalization preprocessing of the time series data includes cleaning, filling missing values ​​and normalization, wherein the missing values ​​in the data are filled using the median by category, and then the processed data is normalized.

[0012] Preferably, the time series data is normalized according to formula (1):

[0013]

[0014] Among them, median is the median, IQR is the interquartile range, Q3 is the upper quartile, and Q1 is the lower quartile.

[0015] Preferably, using a convolutional neural network to extract features from time series data includes: using a convolutional neural network with a convolution kernel of 1 dimension and a size of 5, setting the step size to 1 and the padding to the same during the convolution process, and using ReLU as the activation function, and calculating according to formula (2):

[0016] f(x)=max(0,x) (2)

[0017] Here, x is the input value, which is the output of the previous layer in the neural network.

[0018] Preferably, multiple fully connected layers are introduced to predict each y in the target sequence Y = [y1, y2, y3, y4] i The dimension setting is consistent with the input feature X.

[0019] Preferably, prediction using a cross-attention based Transformer comprises predicting subsequent elements of a sequence using a Transformer decoder, wherein the first layer of the Transformer decoder is a multi-head self-attention layer and the second layer is an encoder-decoder attention layer.

[0020] Preferably, using the cross-attention based Transformer for prediction also includes using dimensional segmented embedding and two-stage attention layers to capture the dependencies between embedded segments, and constructing a hierarchical encoder-decoder; in the encoder layer, except for the first layer, adjacent vectors in the remaining layers are merged and represented, and then a two-stage attention layer is applied to capture the dependencies at this scale; the decoder obtains N feature arrays output by the encoder, and uses N layers for prediction, linearly transforms the output of each layer, and obtains the predicted value of the layer; finally, the predictions of each layer are summed to make a final prediction.

[0021] According to the above technical scheme, the present invention first uses a convolutional neural network to extract features from the input data, and then inputs it into a feature pyramid network for multi-level feature extraction, constructs a series of feature representations of different scales, and fuses features of different scales through a fully connected layer to further refine and enhance the prediction signal, deepen the model's understanding of time series features, and at the same time, promote the learning and integration of cross-layer features, thereby improving the model's prediction performance and generalization ability; finally, a transformer decoder based on cross-attention is used to output the results, including dimensional segmented embedding and a two-stage attention layer, which aims to establish long-distance dependencies, supplement the local structural information omitted by global attention, and make the prediction results more accurate.

[0022] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention but do not constitute a limitation of the present invention. In the accompanying drawings:

[0024] Figure 1 It is a flowchart of the feature pyramid time series prediction method based on cross-attention provided by the present invention. DETAILED DESCRIPTION

[0025] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0026] See also Figure 1 The present invention provides a feature pyramid time series prediction method based on cross attention, the method comprising:

[0027] Normalize and preprocess the time series data;

[0028] Convolutional neural networks are used to extract features from time series data. In addition, in order to reduce the number of features after the convolution operation and avoid overfitting, a pooling operation is performed after each convolution. Since maximum pooling is easily affected by outliers in the features, global average pooling is used.

[0029] The extracted features are extracted using multi-level feature extraction, and the trend features are refined using average pooling. The parameter kernel size is configured to be 3, the step size is 2, and no additional padding is added. The feature pyramid module is used to construct a multi-scale temporal feature representation X.

[0030] After constructing the multi-scale temporal feature representation X, in order to maximize the use of feature information, multiple fully connected layers are introduced to predict the target sequence Y = [y1, y2, y3, y4], where each y in Y = [y1, y2, y3, y4] i The dimension setting of y is consistent with the input feature X, that is, the feature length of each layer is determined by applying the same pooling operation as that used to construct X, ensuring the coordination and consistency of feature dimensions and the effective transmission of information. i With the output y of the previous layer i-1 concatenate, and use the fully connected layer to process so that the output dimension of the fully connected layer is the same as y i Matching aims to fuse feature information of different scales to further refine and enhance the prediction signal;

[0031] Use the cross-attention based Transformer for prediction. In order to enhance cross-dimensional dependencies, the dimension segmentation method is used for embedding. At the same time, the two-stage attention layer is used to capture the dependencies between the segments, and then a hierarchical encoder-decoder is constructed to use information at different scales for final prediction and improve the accuracy of the model. Specifically, the cross-attention based Transformer for prediction also includes using dimension segmentation embedding and two-stage attention layers to capture the dependencies between the embedded segments and construct a hierarchical encoder-decoder; in the encoder layer, except for the first layer, the adjacent vectors in the remaining layers are merged and represented, and then the two-stage attention layer is applied to capture the dependencies at this scale; the decoder obtains the N feature arrays output by the encoder, and uses N layers for prediction, linearly transforms the output of each layer, and obtains the predicted value of the layer; finally, the predictions of each layer are summed to make the final prediction.

[0032] In this embodiment, the normalization preprocessing of the time series data includes cleaning, filling missing values ​​and normalization, wherein the missing values ​​in the data are filled using the median by category, and then the processed data is normalized.

[0033] When performing the above normalization, the time series data is normalized according to formula (1):

[0034]

[0035] Among them, median is the median, IQR is the interquartile range, Q3 is the upper quartile, and Q1 is the lower quartile.

[0036] In this embodiment, using a convolutional neural network to extract features from time series data includes: using a convolutional neural network with a convolution kernel of 1 dimension and a size of 5, setting the step size to 1 and the padding to the same during the convolution process, and using ReLU as an activation function to enhance the nonlinear expression ability of the model, and calculating according to formula (2):

[0037] f(x)=max(0,x) (2)

[0038] Here, x is the input value, which is the output of the previous layer in the neural network.

[0039] In this embodiment, prediction using a cross-attention based Transformer includes predicting subsequent elements of a sequence using a Transformer decoder, wherein the first layer of the Transformer decoder is a multi-head self-attention layer and the second layer is an encoder-decoder attention layer.

[0040] The Transformer decoder predicts subsequent elements of the sequence, and uses multi-head self-attention as the first layer, so that the model can flexibly focus on different positions of the input sequence in each step of generation to achieve high-level integration of information. In order to ensure causality, that is, the output of the current position does not depend on the subsequent information, the decoder adopts a masking strategy to ensure the sequential logic of the generation process; the second layer is the encoder-decoder attention layer, which enables the decoder to directly interact with the output of the encoder, capture and utilize the global context information of the input sequence, and deepen the context understanding ability in the decoding process.

[0041] At the same time, in order to exploit cross-dimensional dependencies, cross attention is used, including dimension segmented embedding and two-stage attention layers, to capture the dependencies between embedded segments. A hierarchical encoder-decoder is constructed using these two modules to utilize information at different scales for final prediction and improve model accuracy.

[0042] A specific implementation is provided below to illustrate the method:

[0043] Step 1: Normalize and preprocess the time series data. The calculation formula is as follows:

[0044]

[0045] Among them, median is the median, IQR is the interquartile range, Q3 is the upper quartile, and Q1 is the lower quartile.

[0046] Step 2: Use a convolutional neural network to extract features from the preprocessed data, perform an average pooling operation after each convolution, and use ReLU as the activation function. The calculation formula is as follows:

[0047] f(x)=max(0,x)

[0048] Here, x is the input value, which is the output of the previous layer in the neural network.

[0049] Step 3: Use multi-level feature extraction to extract the extracted features, use average pooling to refine trend features, and use the feature pyramid module to process and construct a multi-scale time series feature representation X.

[0050] The calculation process is as follows:

[0051] x1=x

[0052] x2=AvgPool(x1)

[0053] x3=AvgPool(x2)

[0054] x4=AvgPool(x3)

[0055] Step 4: For the time series feature representation X, introduce multiple fully connected layers to predict the target sequence Y = [y1, y2, y3, y4], and convert the output y of the current layer i With the output y of the previous layer i-1 concatenate, and use the fully connected layer to process so that the output dimension of the fully connected layer is the same as y i Match, the calculation formula is as follows:

[0056] len(y1)=len(y)

[0057] y′ i-1 =Linear(cat(y′ i ,y i-1 ))

[0058] Step 5: Use the cross-attention based Transformer for prediction, where the cross-attention includes dimensional segment embedding and a two-stage attention layer to capture the dependencies between embedded segments. A hierarchical encoder-decoder is constructed using these two modules to utilize information at different scales for final prediction and improve model accuracy.

[0059] The first is dimensional segmentation embedding, which subdivides the complex high-dimensional space into multiple lower-dimensional subspaces, and then independently calculates the Wasserstein distance on these subspaces, and finally aggregates the distances of all subspaces to obtain the global distance metric of the entire high-dimensional space. The calculation formula is as follows:

[0060] DSW(w)=[seg1(w);seg2(w);…;seg n (w)]

[0061] Among them, w represents the feature, seg i (w) represents the embedding vector of feature w on the i-th dimension segment.

[0062] Two-stage attention deepens understanding through a two-stage attention mechanism, effectively capturing the global and local dependency features in the text, including the global attention processing stage and the local attention refinement stage. The specific implementation steps are summarized as follows:

[0063] In the global attention processing stage, the model assigns a global attention weight to each item in the input sequence, enabling the model to sensitively identify and integrate long-range dependencies in the sequence and build a cross-sentence meaning association framework. The calculation formula is as follows:

[0064]

[0065] in, It is the weight calculated by the attention mechanism.

[0066] In the local attention refinement stage, for each component in the sequence, the attention weight focusing only on the neighboring elements is calculated separately to supplement the local structural information omitted by the global attention. The calculation formula is as follows:

[0067]

[0068] in, is the attention weight of the second stage.

[0069] Finally, the model fuses the global and local attention weights generated in the two stages to form a comprehensive attention distribution that evenly considers the multi-scale features of the text. Based on this fusion weight, a weighted sum operation is performed on each element of the input sequence to generate a new vector that condenses the key information of the text and is input into the decoder for final prediction.

[0070] In summary, the prediction method provided by the present invention provides a feature pyramid time series prediction model based on cross-attention. Through the feature pyramid and cross-attention mechanism, long-distance dependencies are captured, cross-layer feature learning and fusion are promoted, and data of different scales can be processed. In addition, a feature pyramid module is used to extract the extracted features using multi-level features, and the trend features are refined using average pooling. After processing through the feature pyramid module, a series of feature representations of different scales are constructed. The method also uses a Transformer decoder based on cross-attention. In order to enhance cross-dimensional dependencies, a dimensional segmentation method is introduced for embedding. At the same time, a two-stage attention layer is used to capture the dependencies between fragments, and then a hierarchical encoder-decoder is constructed to use information of different scales for final prediction and improve the accuracy of the model.

[0071] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0072] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0073] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0075] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0076] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0077] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0078] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0079] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A feature pyramid time series prediction method based on cross attention, characterized in that: The method comprises: Normalize and preprocess the time series data; Use convolutional neural networks to extract features from time series data and perform a global average pooling operation after each convolution; The extracted features are extracted using multi-level features, and the trend features are refined using average pooling. The output of the previous step is used as the input of the next average pooling. Let x1 = x, and perform an average pooling operation on x1 to obtain x2. Repeat the above operation until x4 is obtained, and use the feature pyramid module to construct a multi-scale temporal feature representation X; Introduce multiple fully connected layers to predict the target sequence Y = [y1, y2, y3, y4], and convert the output y of the current layer i With the output y of the previous layer i-1 concatenate, and use the fully connected layer to process so that the output dimension of the fully connected layer is the same as y i Match; Use the criss-cross attention based Transformer for prediction.

2. The feature pyramid time series prediction method based on cross attention according to claim 1 is characterized in that: Normalization preprocessing of time series data includes cleaning, filling missing values ​​and normalization. For missing values ​​in the data, the median is used to fill them by category, and then the processed data is normalized.

3. The feature pyramid time series prediction method based on cross attention according to claim 2 is characterized in that: The time series data is normalized according to formula (1): Among them, median is the median, IQR is the interquartile range, Q3 is the upper quartile, and Q1 is the lower quartile.

4. The feature pyramid time series prediction method based on cross attention according to claim 1 is characterized in that: Using a convolutional neural network to extract features from time series data includes: using a convolutional neural network with a convolution kernel of 1 dimension and a size of 5, setting the step size to 1 and the padding to the same during the convolution process, and using ReLU as the activation function, and calculating according to formula (2): f(x)=max(0,x) (2) Here, x is the input value, which is the output of the previous layer in the neural network.

5. The feature pyramid time series prediction method based on cross attention according to claim 1 is characterized in that: Introduce multiple fully connected layers to predict each y in the target sequence Y = [y1, y2, y3, y4] i The dimension setting is consistent with the input feature X.

6. The feature pyramid time series prediction method based on cross attention according to claim 1 is characterized in that: Prediction using the criss-cross attention based Transformer involves predicting subsequent elements of a sequence using a Transformer decoder whose first layer is a multi-head self-attention layer and whose second layer is an encoder-decoder attention layer.

7. The feature pyramid time series prediction method based on cross attention according to claim 6 is characterized in that: Prediction using the criss-cross attention based Transformer also involves capturing dependencies between embedded segments using dimension-segment embeddings and two-stage attention layers, and building a hierarchical encoder-decoder; in the encoder layer, adjacent vectors in all layers except the first are merged, and a two-stage attention layer is applied to capture dependencies at this scale; The decoder obtains the N feature arrays output by the encoder and uses N layers to make predictions, linearly transforming the output of each layer to obtain the predicted value of that layer; finally, the predictions of each layer are summed to make the final prediction.

Images