Meteorological feature prediction method based on block and channel fusion
Through the combination of block sequence and multi-head attention mechanism, the problem of long-term dependence modeling of recurrent neural networks and high computational complexity of Transformer model is solved, efficient prediction of climate characteristics is achieved, and prediction accuracy is improved and reduce computational complexity.
Patent Information
- Application Number
- CN202411480502.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-22
- Publication Date
- 2024-12-31
AI Technical Summary
The existing recurrent neural network methods have shortcomings in long-term dependency modeling, which is difficult to effectively capture complex time series patterns, and the computing resources are too large when directly applying the Transformer model to predict climate sequences.
Using a combination of blocking sequence and multi-head attention mechanism, the climate sequence is divided and transformed through linear projection matrix and additive position encoding, and input into the Transformer encoder for processing, using the multi-head attention mechanism and feedforward network layer to reduce the computational complexity. and improve prediction accuracy.
Effectively capture the global features of long-term time series, improve the modeling ability and accuracy of climate feature prediction, and reduce the computational complexity.
Smart Images
Figure CN119227904A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of time series prediction, and in particular to a climate feature prediction method based on a Transformer model, which realizes modeling and prediction of climate data with a long time span through block sequence, position encoding and multi-head attention mechanism. Background Art
[0002] Climate feature prediction is crucial to agricultural production, environmental monitoring, and meteorological warning. Existing recurrent neural network methods are insufficient in modeling long-term dependencies and are difficult to effectively capture complex time series patterns. The Transformer model can handle sequence dependencies globally due to its powerful multi-head attention mechanism, but it consumes too much computing resources when directly applied to climate series prediction. Therefore, an improved method combining block partitioning with Transformer is proposed to reduce computational complexity and improve prediction accuracy. Summary of the invention
[0003] The purpose of the present invention is to provide a meteorological feature prediction method based on block and channel fusion, which can be better in processing complex time dependencies and improving prediction accuracy, while reducing the computational complexity. The technical solution is as follows:
[0004] Step 1: Assume that the original climate series contains L time steps, each time step has M variables, represented as an L×M matrix
[0005] Step 2: Use the last value at the end of the original climate series Repeat filling S times;
[0006] Step 3: Divide the original climate series x into blocks of length P and number of variables M according to the stride of size S, which may or may not overlap. Where P×M is the dimension of each block, is the number of generated blocks;
[0007] Step 4: Using the Linear Projection Matrix and additive positional encoding The block sequence x p Transformed into input sequence x d =W p x p +W pos ,in Where D is the dimension of each input Token;
[0008] Step 5: Transform the sequence x dInput Transformer encoder, obtained through multi-head attention mechanism, query matrix Key Matrix Sum Matrix in and Indicates different attention heads, are the dimensions of the query, key, and value vectors in each attention head;
[0009] Step 6: Calculate the multi-head attention layer output The specific formula is as follows:
[0010]
[0011] Step 7: After calculating the attention output O h After that, a residual connection is made to output the attention O h With input x d Add, and then perform batch normalization, z1 = BatchNorm (x d +O h );
[0012] Step 8: After passing through a feedforward network layer consisting of two fully connected layers, z2 = FFN(z1) = ReLU(W1z1+b1), z3 = W2z2+b2, where W1, W2 are the weight matrices of the feedforward network, and b1, b2 are biases;
[0013] Step 9: The output z3 of the feedforward network is connected to the input z1 of the previous step again through residual connection and batch normalization to obtain the final output z = BatchNorm(z1+z3);
[0014] Step 10: Output After flattening, it passes through a decoder consisting of a fully connected layer to generate prediction results for the next T time steps.
[0015] Step 11: Use the mean square error (MSE) loss function to calculate the loss of each channel using the following formula:
[0016]
[0017] in, represents the predicted value of the i-th time series, Represents the true value of the i-th time series, M represents the number of time series, the loss of each channel of the M time series is summarized, the summarized loss is averaged over the M time series to obtain the overall target loss, and the prediction result is evaluated.
[0018] The beneficial effects of the technical solution provided by the present invention are: using block-based sequence self-attention to effectively capture the global features of long time series, and using the Transformer architecture to improve modeling capabilities, thereby performing better in processing complex time dependencies and improving prediction accuracy, while reducing computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0020] Figure 1 A schematic diagram of a model of a climate feature prediction method based on block and channel fusion provided in an embodiment of the present invention;
[0021] Figure 2 This is the internal flow structure diagram of the Transformer encoder;
[0022] Figure 3 The comparison results of this patent and the current mainstream time series prediction model based on the Transformer architecture;
[0023] Figure 4 A flow chart of a climate feature prediction method based on block and channel fusion provided in an embodiment of the present patent; DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings;
[0025] like Figure 1 As shown, an embodiment of the present invention provides a meteorological feature prediction method based on block and channel fusion, including:
[0026] Step 1: Collect the original climate series containing 96 time steps, each time step contains 21 variables, represented as a 96×21 matrix Among them, 96 is the length of the time series, and 21 is the number of variables in each time step. This matrix is used to store the input climate data. Subsequent steps will be divided and processed based on this matrix;
[0027] Step 2: At the end of the original climate series, use the data from the last time step Repeat filling 8 times to ensure that the entire sequence can be covered in the subsequent block operation and the consistency of the blocks is guaranteed;
[0028] Step 3: Divide the original climate series x into blocks of length 16 and number of variables 21 with a stride of 8 and partial overlap. 16×12 is the dimension of each block, and 12 is the number of generated blocks, ensuring full coverage of the entire input sequence. Through the block operation, a longer time series can be compressed into shorter segments, which helps the model to process long-time span data more effectively.
[0029] Step 4: Using the Linear Projection Matrix and additive positional encoding The block sequence x p Transformed into the input sequence x d =W p x p +W pos ,in Represents the representation of the block sequence in Transformer. 512 is the dimension of each input Token. Linear projection maps each block to a higher-dimensional latent space for further processing by Transformer.
[0030] Step 5: Transform the sequence x d Input Transformer encoder, obtained through multi-head attention mechanism, query matrix Key Matrix Sum Matrix in and Represents different attention heads. 128 is the dimension of query, key and value vectors in each attention head. Four different attention heads are used, and the dimension of the potential representation is 512. The multi-head attention mechanism can capture different patterns of sequences from different subspaces and improve the modeling ability of time series.
[0031] Step 6: Calculate the multi-head attention layer output The specific formula is as follows:
[0032]
[0033] Through the multi-head attention mechanism, the model can process the results of different attention heads in parallel and aggregate the information of each attention head to obtain global dependencies;
[0034] Step 7: After calculating the attention output O h After that, a residual connection is made to output the attention O h With input x d Add, and then perform batch normalization, z1 = BatchNorm (x d +O h), the original input information is retained through residual connections, and the stability of the model is improved through batch normalization;
[0035] Step 8: After passing through the feedforward network layer composed of two fully connected layers, z2=FFN(z1)=ReLU(W1z1+b1), z3=W2z2+b2, where W1, W2 are the weight matrices of the feedforward network, b1, b2 are biases, and the nonlinear activation function ReLU is used to improve the model's ability to express complex relationships;
[0036] Step 9: The output z3 of the feedforward network is connected to the input z1 of the previous step again through residual connection and batch normalization to obtain the final output z=BatchNorm(z1+z3). This step further enhances the generalization ability of the model and ensures the effective capture of long time series information. The process from step 6 to step 9 is as follows: Figure 3 As shown;
[0037] Step 10: Output After flattening, the decoder consists of a fully connected layer to generate predictions for the next 96 time steps. The decoder is used to convert the high-dimensional embedding representation into predictions to predict climate characteristics at future time steps;
[0038] Step 11: Use the mean square error (MSE) loss function to calculate the loss of each channel using the following formula:
[0039]
[0040] in, represents the predicted value of the i-th time series, represents the true value of the i-th time series, 21 represents the number of time series, the loss of each channel of the 21 time series is summarized, the summarized loss is averaged over the 21 time series to obtain the overall target loss, and the prediction results are evaluated. Figure 3 This is the comparison result between this patent and the current mainstream time series prediction model based on the Transformer architecture.
Claims
1. A meteorological feature prediction method based on block and channel fusion, characterized in that The following steps are involved: Step 1: Assume that the original climate series contains L time steps, each time step has M variables, represented as an L×M matrix Step 2: Use the last value at the end of the original climate series Repeat filling S times; Step 3: Divide the original climate series x into blocks of length P and number of variables M according to the stride of size S, which may or may not overlap. Where P×M is the dimension of each block, is the number of generated blocks; Step 4: Using the Linear Projection Matrix and additive positional encoding The block sequence x p Transformed into input sequence x d =W p x p +W pos ,in Where D is the dimension of each input Token; Step 5: Transform the sequence x d Input Transformer encoder, obtained through multi-head attention mechanism, query matrix Key Matrix Sum Matrix in and Indicates different attention heads, are the dimensions of the query, key, and value vectors in each attention head; Step 6: Calculate the multi-head attention layer output The specific formula is as follows: Step 7: After calculating the attention output O h After that, a residual connection is made to output the attention O h With input x d Add, and then perform batch normalization, z1 = BatchNorm (x d +O h ); Step 8: After passing through a feedforward network layer consisting of two fully connected layers, z2 = FFN(z1) = ReLU(W1z1+b1), z3 = W2z2+b2, where W1, W2 are the weight matrices of the feedforward network, and b1, b2 are biases; Step 9: The output z3 of the feedforward network is connected to the input z1 of the previous step again through residual connection and batch normalization to obtain the final output z = BatchNorm(z1+z3); Step 10: Output After flattening, it passes through a decoder consisting of a fully connected layer to generate prediction results for the next T time steps. Step 11: Use the mean square error (MSE) loss function to calculate the loss of each channel using the following formula: in, represents the predicted value of the i-th time series, Represents the true value of the i-th time series, M represents the number of time series, the loss of each channel of the M time series is summarized, the summarized loss is averaged over the M time series to obtain the overall target loss, and the prediction result is evaluated.