Long time series prediction method for hail intensity in area with insufficient features

Abstract

This invention proposes a long-term series prediction method for hail intensity in regions with limited features. It addresses the problems of redundant information and overfitting in traditional time series prediction models due to insufficient input data and features, and the excessive time, space, and memory usage caused by long input data. By improving the Informer model, a long-term series prediction model applicable to the meteorological field is proposed. This method utilizes a multi-granularity attention mechanism based on MPD and a SPH mechanism to prune and eliminate redundant information in the encoder. Furthermore, it employs a ProbSparse self-attention mechanism and self-distillation operations to reduce the time, space, and memory complexity of the encoder. This corrects the redundant information and overfitting issues arising from limited input data and features, thereby improving prediction efficiency and accuracy.

Description

Technical Field

[0001] This invention belongs to the field of time series forecasting technology, and specifically relates to a long-term series forecasting method for hail intensity in areas with insufficient features. Background Technology

[0002] Time series forecasting has been a widely studied and researched topic for many years. With the rapid development of the internet, its application in meteorology has gained widespread recognition. However, current meteorological time series forecasting is limited to short- and medium-term forecasts; further exploration is needed to develop medium- and long-term forecasts.

[0003] Existing methods for solving the LSTF problem include (1) traditional machine learning methods, (2) recurrent neural network (RNN) methods based on RNN architecture, and (3) transformer-based methods.

[0004] However, despite the widespread adoption of transformer architectures [1], [2], we currently lack an understanding of the impact of information patterns within transformer models on long-term sequence prediction tasks. Data in time series prediction tasks are relatively singular because the data is only processed through input embeddings before being input into the model. This characteristic of singularities is amplified in multi-head attention, where the interactions between heads are still poorly understood. Based on this fact, we will start from the following ideas: (1) enhance the expressive power of input data features, (2) enhance the representational power of multi-head attention, and (3) reduce the similarity between multiple attention heads. Below we will introduce the current input data and improvements to the multi-head attention mechanism.

[0005] Regarding data input methods, based on experience, in models with a large number of parameters, the fewer the input data and features, the easier it is to generate redundant information and lead to overfitting [3]. This problem is particularly prominent in transformer structures. For example, the bidirectional encoder representation from the transformer (BERT) model [4] can achieve 0.3B parameters, but BERT does not further process long input sequences except for performing embedding operations (position embedding). Recently, some research has been conducted on using input data processing to enhance model representation. Delight [5] and DeFINE [6] for neural sequence modeling both use linear group transformation and expansion-reduction strategies to process input. However, this strategy greatly increases the computational complexity of the network. MG-SA [7] uses inputs with different attention heads to divide sentences into different granularities for training. The performance of this model has been proven, but the sentence granularization process requires solid prior knowledge or granularization through another model, which is not suitable for LSTF problems.

[0006] Based on experience, the use of multi-head attention mechanisms can improve the accuracy of the resulting model [8]. However, the focus of each attention head is not different. Experiments have shown that some heads can be pruned without affecting the performance of the related model. Paul Michel et al. (2019) [9] pruned heads at different levels and obtained a table containing the number of pruned heads and their impact on the accuracy of the obtained results, which proved that some heads in multi-head attention are redundant. Voita et al. (2019)

[10] performed a redundancy analysis on attention heads by defining the function of attention heads and using confidence and hierarchical correlation propagation (LRP) to quantify the importance of attention heads. The above results show that the heads have specific patterns and are interpretable. However, the functional definition of attention heads requires specific expertise and does not explore the connections between attention heads. CollabHead is used to determine the redundant information between attention heads. An optimized multi-attention head integration method is designed to extract standard information and share it with all attention heads, so that each attention head can focus on capturing unique information, which improves the inference efficiency of BERT, but the method loses some accuracy.

[0007] Pengyu Zeng et al.

[11] found that the similarity values ​​between many attention heads are very high, indicating that different attention heads focus on similar features. Their findings imply that heads can be pruned by comparing similarity values, and that attention inputs can be processed in a multi-granular manner to separate the attention points of each head. To this end, they improved the existing model based on the above viewpoints and established the Muformer model.

[0008] However, in this process, the self-attention mechanism violates the requirement for efficient operation on long sequence inputs and outputs due to its L-quadratic computation and memory consumption on L-length inputs / outputs. Some large-scale Transformer models have poured a lot of resources into NLP tasks and produced impressive results (Brown et al., 2020), but the training on dozens of GPUs and the expensive deployment cost make these models unaffordable for real-world LSTF. Transformer frameworks have become a bottleneck for applying them to solve LSTF problems. Therefore, Haoyi Zhou et al.

[12] proposed the Imformer model: the Transformer model can be improved to increase computational, storage and architectural efficiency while maintaining higher predictive power.

[0009] References:

[0010] [1].LiuY.,PanS.,WangY.G.,XiongF.,WangL.,ChenQ.,LeeV.:Anomalydetec tionindynamicgraphsviatransformer(2021).arXivpreprintarXiv:2106.09876

[0011] [2].LiuZ.,LinY.,CaoY.,HuH.,WeiY.,ZhangZ.,LinS.,GuoB.:Swintransformer:Hierarchicalvisiontransformerusingshiftedwindows(2021).arXivpreprintarXiv:2103.14030

[0012] [3].DaiX.,YinH.,JhaN.K.:NeST:Aneuralnetworksynthesistoolbasedonagrow-and-pruneparadigm.IEEETrans.Comput.,68(10)(2019),pp.1487-1497

[0013] [4].DevlinJ.,ChangM.-W.,LeeK.,ToutanovaK.:Bert:Pre-trainingofdeep bidirectionaltransformersforlanguageunderstanding(2018).arXivpreprint arXiv:1810.04805

[0014] [5].MehtaS.,GhazvininejadM.,IyerS.,ZettlemoyerL.,HajishirziH.:DeLighT:Deepandlight-weighttransformer(2020).arXivpreprintarXiv:2008.00623

[0015] [6].MehtaS.,Koncel-KedziorskiR.,RastegariM.,HajishirziH.:Define:Deepfactorizedinputtokenembeddingsforneuralsequencemodeling(2019).arXivpreprintarXiv:1911.12385

[0016] [7].HaoJ.,WangX.,ShiS.,ZhangJ.,TuZ.:Multi-granularityself-attentionforneuralmachinetranslation(2019).arXivpreprintarXiv:1909.02222

[0017] [8].CordonnierJ.-B.,LoukasA.,JaggiM.:Multi-headattention:Collaborateinsteadofconcatenate(2020).arXivpreprintarXiv:2006.16362

[0018] [9].MichelP.,LevyO.,NeubigG.:Aresixteenheadsreallybetterthanone?(2019).arXivpreprintarXiv:1905.10650

[0019]

[10] .VoitaE.,TalbotD.,MoiseevF.,SennrichR.,TitovI.:Analyzingmulti-headself-attention:Specializedheadsdotheheavylifting,therestcanbepruned(2019).arXivpreprintarXiv:1905.09418

[0020]

[11] .PengyuZeng,GuoliangHu,XiaofengZhou,ShuaiLi,PengjieLiu,ShuruiLiu:Muformer:Alongsequencetime-seriesforecastingmodelbasedonmodifiedmulti-headattention(2022).ISSN0950-7051

[0021]

[12] . Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, Wancai Zhang, Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting Summary of the Invention

[0022] This invention addresses the problems of redundant information and overfitting in traditional time series prediction models due to limited input data and features, and excessive training time complexity, space complexity, and memory usage caused by excessively long input data. It proposes a long-term series prediction model for the meteorological field, utilizing a multi-granularity attention mechanism based on MPD and an inspector model based on SPH to prune and eliminate redundant information in the encoder. Furthermore, it employs a ProbSparse self-attention mechanism and self-distillation operations to reduce the time complexity, space complexity, and memory usage in the encoder, thereby improving prediction efficiency and accuracy.

[0023] To achieve the aforementioned objectives, this invention employs the following technical solution: Based on the Informer model, which includes an encoder and a decoder, the Informer model is improved as follows: The encoder's input receives the historical time series of the target region. The encoder includes an attention module, a slicing module, and a distillation module connected in sequence. The encoder's output is connected to the input of the distillation module, and the decoder's input is connected to the output of the distillation module. The attention module receives the sequence matrix, converts it into a single-head attention, and outputs it to the slicing module. The slicing module receives the sequence matrix and the single-head attention, converts it into a multi-head attention, and outputs it to the distillation module. The distillation module receives the sequence matrix and distills it to obtain a new sequence matrix. The distillation module receives the multi-head attention, trims its attention heads, and obtains a new multi-head attention. The new sequence matrix and the new multi-head attention are output to the decoder, and the decoder outputs the predicted sequence.

[0024] Based on the improved Informer model, historical hail intensity data and historical meteorological information data of the target area within a historical time period are obtained. A dataset containing historical hail intensity sequence data and historical meteorological information of the target area is constructed. The historical meteorological information data includes hail intensity data of the target area to be predicted obtained according to a preset sampling time interval, and meteorological information related to wind power at each hail intensity sampling time, including wind speed, wind direction, temperature, air pressure, and humidity. The improved Informer model is trained using the dataset to obtain a trained hail intensity prediction model, specifically including the following steps:

[0025] Step 1: Obtain historical hail intensity data and historical meteorological information data for the target area within a historical time period, and perform preprocessing and normalization operations to obtain the historical hail intensity time series X for the target area at a preset time interval within the historical time period. t ; For hail intensity time series X t A unified transformation is performed to obtain the hail intensity sequence matrix of the target area over a historical time period. Where t represents the position of the coordinates of each time interval in the time series.

[0026] Specifically, the following steps are included:

[0027] First, historical hail intensity data for the target area at various times within a historical time period are obtained. Then, a historical hail intensity time series X for the target area at preset time intervals within the historical time period is constructed. t , in, Let i be the hail intensity feature vector of the target area at time t within a historical time period, where i∈{1,...,L} x}, L x Represents sequence X t Length, d x This represents the size of the input features, i.e., the number of dimensions of the input features for each time interval. This represents the hail intensity data for the target area at time t;

[0028] Based on an encoder consisting of convolutional and fully connected layers, the time series X of hail intensity is processed. t Input into the convolutional layer; historical hail intensity data at each time step. Perform one-dimensional convolution to obtain the feature vector. The feature vector at each time point is processed by a fully connected layer. The corresponding positional information is used to encode the sequence; the formula for positional encoding is as follows:

[0029]

[0030] in

[0031] Where PE represents the vector being encoded, and t represents the position of the coordinates in the sequence; 2j and 2j+1 represent the 2j and 2j+1th dimensions of the vector, and the dimension being encoded is d_{model}.

[0032] Finally, the 2D coordinate sequence is mapped to the d_{model} dimension and added to the position-encoded vector PE to obtain the hail intensity sequence matrix of the high-dimensional vector with embedded position information. The specific formula is as follows:

[0033]

[0034] Step 2: Obtain the hail intensity sequence matrix The input is fed into the attention module, and the dependency relationship is built in the ProbSparse self-attention mechanism to obtain the single-head attention matrix Attention(Q,K,V);

[0035] Step 2 specifically includes the following steps: generating the hail intensity sequence matrix. The input is fed into the encoder, and its dependencies are built into the ProbSparse self-attention mechanism:

[0036]

[0037]

[0038] Among them, Q, K, and V are based on the hail intensity sequence matrix of the target area over a historical time period. The resulting multi-head attention mechanism layer represents query, key, and value respectively. W is the operation for calculating weights. The weight of the V value is calculated by query Q and key K. After calculating the weights of query Q and key V, the weighted sum of the V values ​​is calculated. d refers to the dimension of the input.

[0039] in,

[0040] The probabilistic form of the attention coefficient for the i-th query is:

[0041]

[0042] To measure the dependency of self-attention units in the encoder corresponding to the input sequence and to quantify the sparsity of the query, KL divergence is used to calculate the relative entropy between the query's attention probability distribution and the uniform probability distribution. The sparsity evaluation formula for the i-th query is as follows:

[0043]

[0044] in, For the Log-Sum-Exp for all keys, It is the arithmetic mean;

[0045] Based on the above evaluation methods, we obtain ProbSparse self-attention, i.e.:

[0046]

[0047] Step 3: Obtain the hail intensity sequence matrix The single-head attention matrix Attention(Q,K,V) is input into the slicing module, and the hail intensity sequence matrix is ​​processed. The single-head attention matrix Attention(Q,K,V) is sliced ​​and input into a multi-granularity attention mechanism based on multi-sensory domain MPD, transforming the single-head attention matrix Attention(Q,K,V) into a multi-head attention MultiHead(Q,K,V) with different granularities; specifically, the following steps are included:

[0048] Hail intensity sequence matrix Perform time block MPD, dividing the time block into N equal-length parts; N is a set hyperparameter, where the MPD mechanism of layer input (1) is as follows: in Different time blocks of a layer slice;

[0049] Local data dependencies are built for each segmented local block by projecting local data. Through learnable matrices Enter different subspaces;

[0050] Then, the similarity between the two spatial vectors and Calculate by dot product and multiply. Reconstruct input data

[0051]

[0052] Set the Nth layer output of MPD to Due to the data in each local block Generate only by establishing dependencies with the data within the block. To enable local block edge data to establish dependencies between surrounding data, a cross-attention method is proposed.

[0053] First, the output is re-sliced, X_Layer 1 First layer:

[0054]

[0055] The proposed MPD-based multi-granularity attention mechanism is modeled by using different heads to process different receptive domains; the output of X_Layer generated by the MPD mechanism is used as the input of different attention heads; where X_Layer is X_Layer={X_Layer j} j∈[N]

[0056] Among them, the total X_Layer j This refers to the output of the j-th layer of the MPD mechanism; the deeper the layer j, the larger the output receptive field; the query input for each head is the same: Q = X_Layer N The key and value are the same in the same header, but different in different headers: K = V = X_Layer j ,j∈[1,…,N];

[0057] head i =Attention(X_Layer N W i Q ,X_Layer j W i K ,X_Layer J W i V )

[0058] in, W is a learnable matrix. i Q W i K W i V Let Q, K, and V represent the linear changes in the head of the i-th attention mechanism layer, respectively.

[0059] After each head has completed its computation, the results of each head are merged, and the output of the multi-head attention mechanism layer is:

[0060] MultiHead(Q,K,V)=Concat(head1,head2,…,head h W o ;

[0061] Among them, W o This represents the weights of the projected linear layer, and the Concat exponential group concatenation operation.

[0062] Step 4: Combine the multi-head attention MultiHead(Q,K,V) and the hail intensity sequence matrix. The input is fed into the distillation module. For the multi-head attention MultiHead(Q,K,V), the attention heads are pruned to eliminate redundant information, resulting in a new multi-head attention MultiHeadnew(Q,K,V). The hail intensity sequence matrix is ​​then processed. Perform distillation to obtain a new sequence matrix. Specifically, the following steps are included:

[0063] For multi-head attention, MultiHead(Q,K,V) achieves regularization by pruning and eliminating redundant information in the head;

[0064] set up The pruning process is as follows:

[0065]

[0066]

[0067]

[0068] Where h represents the nth head, h * The head with the lowest similarity to hyperparameter p;

[0069] Obtain the new multi-head attention MultiHeadnew(Q,K,V):

[0070] MultiHeadnew(Q,K,V)=Concat(head1,head2,...,head h W o

[0071] in,

[0072] For hail intensity sequence matrix Perform a distillation operation on self-attention to extract the main focus and shorten the time dimension of input; the distillation operation from layer j to layer j+1 is as follows:

[0073]

[0074] in,[·] AB This indicates a pruned multi-head self-attention mechanism; Conv1d indicates performing a one-dimensional convolution in the time dimension, where the convolution kernel width is 3 and is followed by the ELU activation function; MaxPool indicates performing max pooling downsampling with a pooling window width of 2, which halves the features of a single layer in the time dimension.

[0075] Step 5: Combine the new multi-head attention function MultiHeadnew(Q,K,V) with the new sequence matrix. The input is fed into the decoder. For the new multi-head attention layer (MultiHeadnew(Q,K,V)) in the encoder, the attention layers of the encoder and decoder interact with features to generate the new sequence matrix. The input is fed into the multi-head attention layer in the decoder, while the input in the decoder is... and Composition of dynamic sampling sequences in, The second half of the encoder input, the sequence When the value is zero, the decoder uses a generative decoding method to obtain the predicted hail intensity sequence Y for the target area at the target time. t Specifically, it includes the following steps:

[0076] will sequence The input is fed into the decoder, and a dynamic sampling sequence is also input into the decoder. in, The starttoken is composed of the second half of the encoder's input sequence. The placeholder reserved for the predicted sequence has a value of zero;

[0077] The entire decoding process of the decoder uses a single forward pass to obtain the final output prediction sequence.

[0078] Step 6: Take the historical hail intensity sequence data and historical meteorological information data of the target area as input, and the corresponding future hail intensity sequence of the target area as output. Use the dataset to train the hail intensity prediction model to be trained, and obtain the trained hail intensity prediction model.

[0079] This invention improves and optimizes the original Informer model, and combines it with the actual needs of hail prediction, enabling more accurate prediction of hail intensity in target areas, with the following beneficial effects:

[0080] 1. Improve prediction accuracy: By preprocessing and normalizing historical hail intensity data and meteorological information data, and by integrating multi-granularity attention mechanisms and dynamic sampling sequences, prediction accuracy can be improved while ensuring computational efficiency.

[0081] 2. Improved prediction range and efficiency: The novel multi-granularity attention mechanism and distillation module proposed in this scheme effectively reduce the computational complexity of the original Informer model, improve the model's running efficiency, and at the same time make the prediction range wider, applicable to more types and scales of hail prediction problems.

[0082] 3. Improve model generalization performance: This solution combines the actual application scenarios of hail prediction and constructs a dataset containing historical hail intensity data and meteorological information data. By training on the dataset, the generalization performance and prediction stability of the model can be improved, and the application scenarios of the model can be cultivated and promoted to more regions and time periods. Attached Figure Description

[0083] Figure 1 This is a flowchart of a long-term series prediction method for hail intensity in areas with insufficient features;

[0084] Figure 2 This is a general block diagram of a long-term series prediction method for hail intensity in areas with insufficient features;

[0085] Figure 3 This is a schematic diagram of the self-distillation mechanism. Detailed Implementation

[0086] The implementation of the technical solution will be further described in detail below with reference to the accompanying drawings:

[0087] like Figure 1-3 As shown, based on the Informer model which includes an encoder and a decoder, the Informer model is improved as follows: The encoder receives the historical time series of the target region as its input. The encoder includes an attention module, a slicing module, and a distillation module connected in sequence. The output of the encoder is connected to the input of the distillation module, and the input of the decoder is connected to the output of the distillation module. The attention module receives the sequence matrix, converts it into a single-head attention, and outputs it to the slicing module. The slicing module receives the sequence matrix and the single-head attention, converts it into a multi-head attention, and outputs it to the distillation module. The distillation module receives the sequence matrix and distills it to obtain a new sequence matrix. The distillation module receives the multi-head attention, trims its attention heads, and obtains a new multi-head attention. The new sequence matrix and the new multi-head attention are output to the decoder, and the decoder outputs the predicted sequence.

[0088] Based on the improved Informer model, historical hail intensity data and historical meteorological information data of the target area within a historical time period are obtained. A dataset containing historical hail intensity sequence data and historical meteorological information of the target area is constructed. The historical meteorological information data includes hail intensity data of the target area to be predicted obtained according to a preset sampling time interval, and meteorological information related to wind power at each hail intensity sampling time, including wind speed, wind direction, temperature, air pressure, and humidity. The improved Informer model is trained using the dataset to obtain a trained hail intensity prediction model, specifically including the following steps:

[0089] Step 1: Obtain historical hail intensity data and historical meteorological information data for the target area within a historical time period, and perform preprocessing and normalization operations to obtain the historical hail intensity time series X for the target area at a preset time interval within the historical time period. t ; For hail intensity time series X t A unified transformation is performed to obtain the hail intensity sequence matrix of the target area over a historical time period. Where t represents the position of the coordinates of each time interval in the time series.

[0090] Specifically, the following steps are included:

[0091] First, historical hail intensity data for the target area at various times within a historical time period are obtained. Then, a historical hail intensity time series X for the target area at preset time intervals within the historical time period is constructed. t , in, Let i be the hail intensity feature vector of the target area at time t within a historical time period, where i∈{1,...,L} x}, L x Represents sequence X t Length, d x This represents the size of the input features, i.e., the number of dimensions of the input features for each time interval. This represents the hail intensity data for the target area at time t;

[0092] Based on an encoder consisting of convolutional and fully connected layers, the time series X of hail intensity is processed. t Input into the convolutional layer; historical hail intensity data at each time step. Perform one-dimensional convolution to obtain the feature vector. The feature vector at each time point is processed by a fully connected layer. The corresponding positional information is used to encode the sequence; the formula for positional encoding is as follows:

[0093]

[0094] in

[0095] Where PE represents the vector being encoded, and t represents the position of the coordinates in the sequence; 2j and 2j+1 represent the 2j and 2j+1th dimensions of the vector, and the dimension being encoded is d_{model}.

[0096] Finally, the 2D coordinate sequence is mapped to the d_{model} dimension and added to the position-encoded vector PE to obtain the hail intensity sequence matrix of the high-dimensional vector with embedded position information. The specific formula is as follows:

[0097]

[0098] Step 2: Obtain the hail intensity sequence matrix The input is fed into the attention module, and the dependency relationship is built in the ProbSparse self-attention mechanism to obtain the single-head attention matrix Attention(Q,K,V);

[0099] Step 2 specifically includes the following steps: generating the hail intensity sequence matrix. The input is fed into the encoder, and its dependencies are built into the ProbSparse self-attention mechanism:

[0100]

[0101]

[0102] Among them, Q, K, and V are based on the hail intensity sequence matrix of the target area over a historical time period. The resulting multi-head attention mechanism layer represents query, key, and value respectively. W is the operation for calculating weights. The weight of the V value is calculated by query Q and key K. After calculating the weights of query Q and key V, the weighted sum of the V values ​​is calculated. d refers to the dimension of the input.

[0103] in,

[0104] The probabilistic form of the attention coefficient for the i-th query is:

[0105]

[0106] To measure the dependency of self-attention units in the encoder corresponding to the input sequence and to quantify the sparsity of the query, KL divergence is used to calculate the relative entropy between the query's attention probability distribution and the uniform probability distribution. The sparsity evaluation formula for the i-th query is as follows:

[0107]

[0108] in, For the Log-Sum-Exp for all keys, It is the arithmetic mean;

[0109] Based on the above evaluation methods, we obtain ProbSparse self-attention, i.e.:

[0110]

[0111] Step 3: Obtain the hail intensity sequence matrix The single-head attention matrix Attention(Q,K,V) is input into the slicing module, and the hail intensity sequence matrix is ​​processed. The single-head attention matrix Attention(Q,K,V) is sliced ​​and input into a multi-granularity attention mechanism based on multi-sensory domain MPD, transforming the single-head attention matrix Attention(Q,K,V) into a multi-head attention MultiHead(Q,K,V) with different granularities; specifically, the following steps are included:

[0112] Hail intensity sequence matrix Perform time block MPD, dividing the time block into N equal-length parts; N is a set hyperparameter, where the MPD mechanism of layer input (1) is as follows: in Different time blocks of a layer slice;

[0113] Local data dependencies are built for each segmented local block by projecting local data. Through learnable matrices Enter different subspaces;

[0114] Then, the similarity between the two spatial vectors and Calculate by dot product and multiply. Reconstruct input data

[0115]

[0116] Set the Nth layer output of MPD to Each local block's data Xi1 can only be generated by establishing dependencies with the data within the block. However, since local edge data cannot establish dependencies between surrounding data, and some data also have long-sequence dependency requirements, a cross-attention method is proposed to address this issue.

[0117] First, the output is re-sliced, X_Layer 1 First layer:

[0118]

[0119] The input is a new local block to be constructed, containing edge data from the previous layer for which dependencies have not yet been established. Through an attention mechanism, dependencies on the data within the local block are built, solving the problem of edge data in local blocks lacking dependencies. Furthermore, since the local blocks from the previous layer overlap with the current local block, this block is essentially connected to adjacent local blocks from the previous layer. When the MPD mechanism overlaps with four layers of local blocks, the data's receptive field is amplified, resolving some long-range dependency issues. The closer the MPD output data is to the center, the larger the data's receptive field; the closer the data is to the prediction point, the smaller the data's receptive field. Different MPD output data layers have different receptive fields. The deeper the layer, the larger the receptive area near the center and the smaller the receptive area near the prediction point. These operations expand the data's receptive field and reduce information loss around the prediction point due to information transmission.

[0120] The proposed multi-granularity attention mechanism based on MPD is modeled by different heads processing different receptive domains. The output of X_Layer (N=4) generated by the MPD mechanism is used as the input to different attention heads. Here, X_Layer is:

[0121] X_Layer is X_Layer = {X_Layer} j} j∈[N]

[0122] Among them, the total X_Layer j This refers to the output of the j-th layer of the MPD mechanism; the deeper the layer j, the larger the output receptive field; the query input for each head is the same: Q = X_Layer N The key and value are the same in the same header, but different in different headers: K = V = X_Layer j ,j∈[1,...,N];

[0123]

[0124] in, W is a learnable matrix. i Q W i K W i V Let Q, K, and V represent the linear changes in the head of the i-th attention mechanism layer, respectively.

[0125] After each head has completed its computation, the results of each head are merged, and the output of the multi-head attention mechanism layer is:

[0126] MultiHead(Q,K,V)=Concat(head1,head2,...,headh W o ;

[0127] Among them, W o This represents the weights of the projected linear layer, and the Concat exponential group concatenation operation.

[0128] Step 4: Combine the multi-head attention MultiHead(Q,K,V) and the hail intensity sequence matrix. The input is fed into the distillation module. For the multi-head attention MultiHead(Q,K,V), the attention heads are pruned to eliminate redundant information, resulting in a new multi-head attention MultiHeadnew(Q,K,V). The hail intensity sequence matrix is ​​then processed. Perform distillation to obtain a new sequence matrix. Specifically, the following steps are included:

[0129] For multi-head attention, MultiHead(Q,K,V) achieves regularization by pruning and eliminating redundant information in the head;

[0130] set up The pruning process is as follows:

[0131]

[0132]

[0133]

[0134] Where h represents the nth head, h * The head with the lowest similarity to hyperparameter p;

[0135] Obtain the new multi-head attention MultiHeadnew(Q,K,V):

[0136] MultiHeadnew(Q,K,V)=Concat(head1,head2,...,head h W o

[0137] in,

[0138] For hail intensity sequence matrix Perform a distillation operation on self-attention to extract the main focus and shorten the time dimension of input; the distillation operation from layer j to layer j+1 is as follows:

[0139]

[0140] in,[·] ABThis indicates a pruned multi-head self-attention mechanism; Conv1d indicates performing a one-dimensional convolution in the time dimension, where the convolution kernel width is 3 and is followed by the ELU activation function; MaxPool indicates performing max pooling downsampling with a pooling window width of 2, which halves the features of a single layer in the time dimension.

[0141] Step 5: Combine the new multi-head attention function MultiHeadnew(Q,K,V) with the new sequence matrix. The input is fed into the decoder. For the new multi-head attention layer (MultiHeadnew(Q,K,V)) in the encoder, the attention layers of the encoder and decoder interact with features to generate the new sequence matrix. The input is fed into the multi-head attention layer in the decoder, while the input in the decoder is... and Composition of dynamic sampling sequences in, The second half of the encoder input, the sequence When the value is zero, the decoder uses a generative decoding method to obtain the predicted hail intensity sequence Y for the target area at the target time. t Specifically, it includes the following steps:

[0142] will sequence The input is fed into the decoder, and a dynamic sampling sequence is also input into the decoder. in, The starttoken is composed of the second half of the encoder's input sequence. The placeholder reserved for the predicted sequence has a value of zero; the entire decoder's decoding process abandons the dynamic decoding process and uses a single forward pass to decode the entire output sequence. The decoder receives a long sequence input, fills the target elements with zeros, measures the weighted attention composition of the feature map, and immediately predicts the output elements in a generative manner.

[0143]

[0144] Step 6: Take the historical hail intensity sequence data and historical meteorological information data of the target area as input, and the corresponding future hail intensity sequence of the target area as output. Use the dataset to train the hail intensity prediction model to be trained, and obtain the trained hail intensity prediction model.

[0145] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0146] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention.

[0147] Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A long-term series prediction method for hail intensity in areas with insufficient features, characterized in that, Based on the Informer model, which includes an encoder and a decoder, the Informer model is improved as follows: The encoder receives the historical time series of the target region as its input. The encoder includes an attention module, a slicing module, and a distillation module connected in sequence. The output of the encoder is connected to the input of the distillation module, and the input of the decoder is connected to the output of the distillation module. The attention module receives the sequence matrix, converts it into a single-head attention, and outputs it to the slicing module. The slicing module receives the sequence matrix and the single-head attention, converts it into a multi-head attention, and outputs it to the distillation module. The distillation module receives the sequence matrix, distills it to obtain a new sequence matrix, receives the multi-head attention, trims the attention heads, and obtains a new multi-head attention. The new sequence matrix and the new multi-head attention are output to the decoder, and the decoder outputs the predicted sequence. Based on the improved Informer model, historical hail intensity data and historical meteorological information data of the target area within a historical time period are obtained. A dataset containing historical hail intensity sequence data and historical meteorological information data of the target area is constructed. The improved Informer model is trained using the dataset to obtain a trained hail intensity prediction model. The specific steps include: Step 1: Obtain historical hail intensity data and historical meteorological information data for the target area within a historical time period, and perform preprocessing and normalization operations to obtain the historical hail intensity time series X for the target area at a preset time interval within the historical time period. t ; For hail intensity time series X t A unified transformation is performed to obtain the hail intensity sequence matrix of the target area over a historical time period. Where t represents the position of the coordinates of each time interval in the time series; Step 2: Obtain the hail intensity sequence matrix The input is fed into the attention module, and the dependency relationship is built in the ProbSparse self-attention mechanism to obtain the single-head attention matrix Attention(Q,K,V); Step 3: Obtain the hail intensity sequence matrix The single-head attention matrix Attention(Q,K,V) is input into the slicing module, and the hail intensity sequence matrix is ​​processed. The slices are then input into a multi-granularity attention mechanism based on multi-sensory domain MPD, transforming the single-head attention matrix Attention(Q,K,V) into a multi-head attention MultiHead(Q,K,V) with different granularities. Step 4: Combine the multi-head attention MultiHead(Q,K,V) and the hail intensity sequence matrix. The input is fed into the distillation module. For the multi-head attention MultiHead(Q,K,V), the attention heads are pruned to eliminate redundant information, resulting in a new multi-head attention MultiHeadnew(Q,K,V). The hail intensity sequence matrix is ​​then processed. Perform distillation to obtain a new sequence matrix. Step 5: Combine the new multi-head attention function MultiHeadnew(Q,K,V) with the new sequence matrix. The input is fed into the decoder. For the new multi-head attention layer (MultiHeadnew(Q,K,V)) in the encoder, the attention layers of the encoder and decoder interact with features to generate the new sequence matrix. The input is fed into the multi-head attention layer in the decoder, while the input in the decoder is... and Composition of dynamic sampling sequences in, The second half of the encoder input, the sequence When the value is zero, the decoder uses a generative decoding method to obtain the predicted hail intensity sequence Y for the target area at the target time. t ; Step 6: Take the historical hail intensity sequence data and historical meteorological information data of the target area as input, and the corresponding future hail intensity sequence of the target area as output. Use the dataset to train the hail intensity prediction model to be trained, and obtain the trained hail intensity prediction model.

2. The long-term series prediction method for hail intensity in areas with insufficient features according to claim 1, characterized in that... Step 1 specifically includes the following steps: First, historical hail intensity data for the target area at various times within a historical time period are obtained. Then, a historical hail intensity time series X for the target area at preset time intervals within the historical time period is constructed. t , in, Let i be the hail intensity feature vector of the target area at time t within a historical time period, where i∈{1,...,L} x }, L x Represents sequence X t Length, d x This represents the size of the input features, i.e., the number of dimensions of the input features for each time interval. This represents the hail intensity data for the target area at time t; Based on an encoder consisting of convolutional and fully connected layers, the time series X of hail intensity is processed. t Input into the convolutional layer; historical hail intensity data at each time step. Perform one-dimensional convolution to obtain the feature vector. The feature vector at each time point is processed by a fully connected layer. The corresponding positional information is used to encode the sequence; the formula for positional encoding is as follows: in Where PE represents the vector being encoded, and t represents the position of the coordinates in the sequence; 2j and 2j+1 represent the 2j and 2j+1th dimensions of the vector, and the dimension being encoded is d_{model}. Finally, the 2D coordinate sequence is mapped to the d_{model} dimension and added to the position-encoded vector PE to obtain the hail intensity sequence matrix of the high-dimensional vector with embedded position information. The specific formula is as follows:

3. The long-term series prediction method for hail intensity in areas with insufficient features according to claim 1, characterized in that, Step 2 specifically includes the following steps: Hail intensity sequence matrix The input is fed into the encoder, and its dependencies are built into the ProbSparse self-attention mechanism: Among them, Q, K, and V are based on the hail intensity sequence matrix of the target area over a historical time period. The resulting multi-head attention mechanism layer represents query, key, and value respectively. W is the operation for calculating weights. The weight of the V value is calculated by query Q and key K. After calculating the weights of query Q and key V, the weighted sum of the V values ​​is calculated. d refers to the dimension of the input. in, The probabilistic form of the attention coefficient for the i-th query is: To measure the dependency of self-attention units in the encoder corresponding to the input sequence and to quantify the sparsity of the query, KL divergence is used to calculate the relative entropy between the query's attention probability distribution and the uniform probability distribution. The sparsity evaluation formula for the i-th query is as follows: in, For the Log-Sum-Exp for all keys, It is the arithmetic mean; Based on the above evaluation methods, we obtain ProbSparse self-attention, i.e.:

4. The long-term series prediction method for hail intensity in areas with insufficient features according to claim 1, characterized in that, Step 3 specifically includes the following steps: Hail intensity sequence matrix Perform time block MPD, dividing the time block into N equal-length parts; N is a set hyperparameter, where the MPD mechanism of layer input (1) is as follows: in Different time blocks of a layer slice; Local data dependencies are built for each segmented local block by projecting local data. Through learnable matrices Enter different subspaces; Then, the similarity between the two spatial vectors and Calculate by dot product and multiply by V i (1) Reconstruct the input data Set the Nth layer output of MPD to Due to the data in each local block Generate only by establishing dependencies with the data within the block. To enable local block edge data to establish dependencies between surrounding data, a cross-attention method is proposed. First, the output is re-sliced, X_Layer 1 First layer: The proposed MPD-based multi-granularity attention mechanism is modeled by using different heads to process different receptive domains; the output of X_Layer generated by the MPD mechanism is used as the input of different attention heads; where X_Layer is X_Layer={X_Layer j } j∈[N] Among them, the total X_Layer j This refers to the output of the j-th layer of the MPD mechanism; the deeper the layer j, the larger the output receptive field; the query input for each head is the same: Q = X_Layer N The key and value are the same in the same header, but different in different headers: K = V = X_Layer j ,j∈[1,...,N]; head i =Attention(X_Layer N W i Q ,X_Layer j W i K ,X_Layer J W i V ) in, W is a learnable matrix. i Q W i K W i V Let Q, K, and V represent the linear changes in the head of the i-th attention mechanism layer, respectively. After each head has completed its computation, the results of each head are merged, and the output of the multi-head attention mechanism layer is: MultiHead(Q,K,V)=Concat(head1,head2,...,head h )W o ; Among them, W o This represents the weights of the projected linear layer, and the Concat exponential group concatenation operation.

5. The long-term series prediction method for hail intensity in areas with insufficient features according to claim 1, characterized in that, Step 4 specifically includes the following steps: For multi-head attention, MultiHead(Q,K,V) achieves regularization by pruning and eliminating redundant information in the head; set up The pruning process is as follows: Where h represents the nth head, h * The head with the lowest similarity to hyperparameter p; Obtain the new multi-head attention MultiHeadnew(Q,K,V): MultiHeadnew(Q,K,V)=Concat(head1,head2,...,head h )W o in, 6. The long-term series prediction method for hail intensity in areas with insufficient features according to claim 1, characterized in that, Step 4 specifically includes the following steps: For hail intensity sequence matrix Perform a distillation operation on self-attention to extract the main focus and shorten the time dimension of input; the distillation operation from layer j to layer j+1 is as follows: in,[·] AB This indicates a pruned multi-head self-attention mechanism; Conv1d indicates performing a one-dimensional convolution in the time dimension, where the convolution kernel width is 3 and is followed by the ELU activation function; MaxPool indicates performing max pooling downsampling with a pooling window width of 2, which halves the features of a single layer in the time dimension.

7. The long-term series prediction method for hail intensity in areas with insufficient features according to claim 1, characterized in that, Step 5 specifically includes the following steps: will sequence The input is fed into the decoder, and a dynamic sampling sequence is also input into the decoder. in, The start token is composed of the second half of the sequence extracted from the encoder's input. The placeholder reserved for the predicted sequence has a value of zero; The entire decoding process of the decoder uses a single forward pass to obtain the final output prediction sequence.

8. The long-term series prediction method for hail intensity in areas with insufficient features according to claim 1, characterized in that, The historical meteorological information data includes hail intensity of the target area to be predicted obtained according to a preset sampling time interval, meteorological information related to wind power at each hail intensity sampling time, and meteorological information related to wind power including wind speed, wind direction, temperature, air pressure, and humidity.

Images