Spacetime lstm network based on self-attention mechanism for radar echo sequence prediction method

Abstract

The application discloses a spatio-temporal LSTM network radar echo sequence prediction method based on a self-attention mechanism, and specifically comprises the following steps: dividing a CKIM radar echo dataset into a training set and a test set, and performing pretreatment; adopting a self-attention mechanism to replace a forgetting gate mechanism in an ST-LSTM unit to form an SA-ST-LSTM unit; building an encoding-attention-decoding network; feeding the training set into the encoding-attention-decoding network to perform training, and obtaining a training model; feeding the test set into the training model to perform testing, and obtaining an image prediction result and prediction data. The SA-ST-LSTM unit is proposed, and the encoding-attention-decoding network is designed, the regulation of hyperparameters on long-term memory and short-term memory is adopted to process a catastrophic forgetting problem in the forgetting gate, and in addition, the attention mechanism is added to delay a long-term memory gradual forgetting problem in the network, so that the performance of radar echo sequence prediction is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of radar echo sequence prediction, and particularly relates to a spatio-temporal LSTM network radar echo sequence prediction method based on a self-attention mechanism. BACKGROUND

[0002] Precipitation nowcasting is an indispensable part of people's daily life, and is of great significance to the decision-making of various industries and society. Effective precipitation prediction can better guide people's daily work and life. The current precipitation nowcasting technology mainly realizes the prediction of precipitation intensity and area by judging the movement trajectory of the atmosphere and airflow. Among them, short-term nowcasting is the main research focus. The future sequence frame is predicted by collecting historical frame radar echo sequence data, so as to judge the precipitation nowcasting situation by using the intensity and area of the future frame. The current main method is to use historical radar sequence data for training to construct a pre-training model, and to predict the future frame by using the pre-training model, judge the precipitation situation according to the prediction result, and further improve the accuracy of radar echo sequence prediction by continuously improving the structure of the deep learning network. These studies expand the understanding of precipitation nowcasting and provide a new way to improve prediction accuracy and decision support capability. SUMMARY

[0003] The purpose of the present application is to provide a spatio-temporal LSTM network radar echo sequence prediction method based on a self-attention mechanism to improve the short-term nowcasting radar echo sequence prediction capability.

[0004] The technical solution adopted by the present application is a spatio-temporal LSTM network radar echo sequence prediction method based on a self-attention mechanism, which is implemented according to the following steps:

[0005] Step 1, divide the CKIM radar echo dataset into a training set and a test set, and preprocess to obtain sequence images;

[0006] Step 2, replace the forgetting gate mechanism in the ST-LSTM unit with a self-attention mechanism to form a SA-ST-LSTM unit;

[0007] Step 3, use the SA-ST-LSTM unit to build an encoding-attention-decoding network;

[0008] Step 4, input the training set into the encoding-attention-decoding network for training to obtain an encoding-attention-decoding training model;

[0009] Step 5, input the test set into the encoding-attention-decoding training model for testing to obtain the image prediction result and prediction data of the future frame.

[0010] The application is also characterized in that

[0011] In step 1, the training set of the CKIM radar echo data set contains 120,000 images, the test set contains 30,000 images, the training set contains 24,000 sequences, and the test set contains 6,000 sequences; the size of all images in the two data sets is adjusted to 101*101*1, and normalization processing is performed to obtain sequence images.

[0012] In step 2, specifically:

[0013] In the ST-LSTM unit, due to the over-saturation of the forgetting gate, the long-term memory C t and the short-term memory H t cannot be effectively transmitted, the forgetting gate is replaced, and the self-attention mechanism is introduced to aggregate features, important features are collected, and redundant information is discarded, and the attention mechanism is implemented as follows: the long-term memory state C t-1 and the hidden state H t of the previous moment are used, and then the hyperparameters alpha and 1-alpha are set to control the amount of information transmission, so as to realize the timely update of information, and the formula of the finally obtained SA-ST-LSTM unit is as follows:

[0014]

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] In the formula, t represents the time step, l represents the number of stacked layers, C t-1 represents the long-term memory at t moment, C i represents the long-term memory at t-1 moment, alpha is an adjustable hyperparameter, x t-1 is an input state, h t is a short-term memory state at t-1 moment, g' t , t respectively represent the modulation gate, the input gate and the forgetting gate; is the space-time memory of the l-1 layer at t moment, sigma is a sigmoid function, and tanh is a tanh function. is the space-time memory of the l-th layer at time t; e represents a vector; o t is the input of the l-th layer at time t

[0022] Go out; * represents Hadamard convolution; attention is a self-attention mechanism; b g ,b i ,b f ,b x ,W xg ,W mg ,W xi ,W mi ,W xf ,W mf ,W xo ,W ho ,W co ,W mo ,W 1×1 are all weight matrices.

[0023] In step 3, specifically:

[0024] A 3-layer CNN and a 3-layer SA-ST-LSTM unit are used for cross-stacking to form an encoding-decoding structure, wherein in the horizontal direction, the long-term memory state C t and the hidden state H t of the SA-ST-LSTM unit are transmitted along the horizontal direction, and in the vertical direction, the short-term memory state H t and the space-time memory unit M t are transmitted as the input of the next layer, and the space-time memory unit M t is continuous in the transmission process of the entire network, and secondly, an attention mechanism is added after encoding to further extract important information from the long-term memory C t , the short-term memory H t , and the space-time memory M t obtained after encoding, and the attention mechanism is as follows:

[0025]

[0026] wherein, represents the long-term memory state the short-term memory state and the current space-time memory state of the l-th layer, respectively, represent the short-term memory state, the long-term memory state, and the current space-time memory state of the l-th layer after the attention mechanism;

[0027] The 3-layer SA-ST-LSTM unit and the 3-layer CNN are stacked to form a decoding structure, and the input to the decoding structure is completed decoding through an attention mechanism The decoded result is finally stacked and feature fusion is performed through a 1*1 convolution kernel to obtain the obtained encoding-attention-decoding network.

[0028] In step 4, specifically: the training set is input as a sequence of 5 consecutive frames, and 10 frames are used as a sequence of true values, and the pre-trained model converged on the data set is obtained by optimizing the MSE loss function, and the radar echo is predicted by the pre-trained model to determine the precipitation condition.

[0029] The present application has the advantages that: the present application proposes the SA-ST-LSTM unit and designs the encoding-attention-decoding network, compared with the traditional ST-LSTM, the SA-ST-LSTM unit introduces the attention mechanism to replace the forgetting gate mechanism, and the long-term memory and the short-term memory are adjusted by using the hyperparameter to process the catastrophic forgetting problem in the forgetting gate; the encoding-attention-decoding network is proposed on the encoding-decoding network, the network adopts the cross mode of the CNN and the SA-ST-LSTM unit to realize the effective extraction of the features, the attention mechanism is added to delay the long-term memory forgetting problem in the encoding-decoding network, and the performance of the radar echo sequence prediction is improved. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is the calculation process diagram of the self-attention mechanism in the radar echo sequence prediction method based on the self-attention mechanism of the space-time LSTM network of the present application;

[0031] Figure 2 is the SA-ST-LSTM unit attention internal module diagram in the radar echo sequence prediction method based on the self-attention mechanism of the space-time LSTM network of the present application;

[0032] Figure 3 is the SA-ST-LSTM unit structure diagram in the radar echo sequence prediction method based on the self-attention mechanism of the space-time LSTM network of the present application;

[0033] Figure 4 is the encoding-attention-decoding network structure diagram in the radar echo sequence prediction method based on the self-attention mechanism of the space-time LSTM network of the present application.

[0034] Figure 5 is the radar echo prediction diagram of the CKIM data set in the implementation of the present application;

[0035] Figure 6 is the ten-frame index diagram of the precipitation prediction of the HSS threshold value τ of 30 in the implementation of the present application;

[0036] Figure 7 is the ten-frame index graph of the precipitation prediction in the implementation of the application, and the CSI threshold τ is 30;

[0037] Figure 8 is a graph showing the influence of the super parameter α setting on the SA-ST-LSTM unit in the implementation of the application;

[0038] Figure 9 is a radar echo graph visualizing the α value in different SA-ST-LSTM units in the implementation of the application. DETAILED DESCRIPTION

[0039] The application will be described in detail below in combination with the drawings and specific embodiments.

[0040] The radar echo sequence prediction method based on the self-attention mechanism time-space LSTM network of the application is implemented according to the following steps.

[0041] Step 1, divide the CKIM radar echo dataset into a training set and a test set, preprocess the dataset, and obtain sequence images;

[0042] Specifically, the training set of the CKIM radar echo dataset contains 120,000 images, and the test set contains 30,000 images. The training set contains 24,000 sequences, and the test set contains 6,000 sequences. Select 5 frames as input and 10 frames as output for each sequence. Adjust the size of all images in the two datasets to 101x101x1, and perform normalization processing to obtain sequence images.

[0043] Step 2, improve the ST-LSTM unit, replace the forgetting gate mechanism in the ST-LSTM unit with a self-attention mechanism, and adjust the long-term memory and short-term memory state update by adjusting the super parameter α, so as to solve the catastrophic forgetting problem of the ST-LSTM unit during the prediction process and reduce the gradient disappearance phenomenon caused by the inability of the ST-LSTM unit to update information during transmission.

[0044] The self-attention module is proposed for the global dependency of the input. In the image, the self-attention module calculates the pairwise relationship between different positions of the feature map through a binary relationship function to capture long-range space-time dependency relationships. Then, the features of interest are calculated through these relationships. The self-attention module usually uses clicks to calculate attention scores. Here, the dimensions of the query vector and the key vector are the same. Figure 1 The calculation process of the self-attention mechanism used is shown. The original feature C t is mapped to different feature spaces, where the query is the key is and the value is The convolution kernel size used is 1x1, and the last C t and The number of channels is the same. First, the similar attention score is calculated by matrix multiplication operation, which is calculated as follows:

[0045]

[0046] The expression of Softmax is as follows:

[0047]

[0048] In the ST-LSTM unit, due to the over-saturation of the forgetting gate, the long-term memory C t and the short-term memory H t cannot be effectively transmitted, the forgetting gate is replaced, and the self-attention mechanism is introduced to aggregate the features. After gathering important features, the redundant information is discarded, so that the information of the model can be better transmitted and its spatiotemporal correlation is improved. The implementation of the combined attention mechanism is shown in Figure 2 The long-term memory state C t-1 and the hidden state H t of the previous time are used to pay more attention to the information needed, and then the hyperparameters a and 1-a are set to control the amount of information transmission, so as to realize the timely update of information. The structure of the SA-ST-LSTM unit is shown in Figure 3 The formula of the SA-ST-LSTM unit is as follows:

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

[0056] In the formula, t represents the time step, l represents the number of stacked layers, C t-1 represents the long-term memory at t time; C i represents the long-term memory at t-1 time, a is an adjustable hyperparameter; x t-1 is the input state; h ti', i', i' respectively represent modulation gate, input gate and forget gate; t ,f t i', i', i' respectively represent modulation gate, input gate and forget gate; is the space-time memory of the l-1 layer at t time; σ is Sigmoid function; tanh is tanh function; is the space-time memory of the l layer at t time; e represents vector; o t is output gate; * represents Hadamard convolution; attention is self-attention mechanism; b′ g ,b′ i ,b′ f ,b0are bias terms, W x ,W' xg ,W mg ,W' xi ,W mi ,W' xf ,W mf ,W xo ,W ho ,W co ,W mo ,W 1×1 are weight matrices.

[0057] Step 3, using SA-ST-LSTM unit to build encoding-attention-decoding network;

[0058] Further extraction of features is realized through cross-stacking of CNN and SA-ST-LSTM unit, secondly, attention mechanism is added to the encoding-decoding structure, which delays the gradual forgetting phenomenon of long-term memory in the decoding process, and finally forms the encoding-attention-decoding network;

[0059] Specifically: 3 layers of CNN and 3 layers of SA-ST-LSTM unit are cross-stacked to form an encoding-decoding structure, wherein in the horizontal direction, the long-term memory state C t and the hidden state H t of the SA-ST-LSTM unit are transmitted along the horizontal direction, in the vertical direction, the short-term memory state H t and the space-time memory unit M t are transmitted as the input of the next layer, and the space-time memory unit M t is continuous in the whole network transmission process, which forms a unified space-time change structure in time and space, secondly, attention mechanism is added after encoding, so that the long-term memory C t , short-term memory H t and space-time memory M t of the vector obtained after encoding are further extracted important information through attention mechanism, and the expression of the attention mechanism for feature extraction after encoding is as follows:

[0060]

[0061] wherein, represents the long-term memory state of the lth layer represents the short-term memory state of the lth layer and the current spatio-temporal memory state respectively represent the short-term memory state, the long-term memory state and the current spatio-temporal memory state of the lth layer after attention mechanism. The 3-layer SA-ST-LSTM unit and the 3-layer CNN are stacked to form a decoding structure, and the short-term memory state, the long-term memory state and the current spatio-temporal memory state after attention mechanism are input into the decoding structure to complete decoding, and finally the decoding results are stacked and feature fusion is performed using a 1x1 convolution kernel to obtain the obtained encoding-attention-decoding network, as shown in Figure 4 .

[0062] In the encoding-decoding structure of radar echo sequence prediction, the spatio-temporal sequence unit is stacked, and the ability of the spatio-temporal sequence unit to capture spatial information is not as powerful as the convolution layer, therefore, after adding a convolution layer for feature extraction, the network can further improve the ability to capture spatial information, and obtain a clearer predicted image. Secondly, since the long-term memory information will usually have a gradual forgetting trend during decoding, in order to further enhance the ability to capture important information, an attention mechanism is added in the encoding and decoding process, so that the long-term memory C t , the short-term memory H t and the spatio-temporal memory M t of the vector obtained after encoding all pass through the attention mechanism to further extract important information;

[0063] Step 4, input the training set into the encoding-attention-decoding network for training to obtain an encoding-attention-decoding training model;

[0064] The training set is input in a sequence of 5 frames, and 10 frames are used as a sequence of true values, and the MSE loss function is used for optimization, so as to obtain a pre-trained model converged on the data set. The pre-trained model is used to realize the prediction of the radar echo, so as to judge the precipitation condition.

[0065] Step 5, input the test set into the encoding-attention-decoding training model for testing to obtain test data and predicted images;

[0066] The test set is input into the training model in a sequence for testing, and the CKIM radar echo data set is used to test the final prediction effect. Finally, the output result is converted into an image for saving to obtain the image prediction result and prediction data of the future frame.

[0067] Embodiment

[0068] The SA-ST-LSTM unit composed of the encoding-attention-decoding network is evaluated using the CKIM radar echo dataset. In the analysis, the EnADe (ST-LSTM) and EnADe (SA-ST-LSTM) structures represent the encoding-attention-decoding network based on the ST-LSTM unit and the encoding-attention-decoding network based on the SA-ST-LSTM unit, the data experimental results and the predicted image results of the network are analyzed, and the ablation experiment of the weather prediction image is passed, and the value of a in the SA-ST-LSTM unit is analyzed. The square loss function is used to train and test the network, the experimental results of the CKIM radar echo and the predicted image are carried out under the hyperparameter a of the SA-ST-LSTM unit being 0.7, all experiments are implemented in Pytorch and carried out on the RTX 3080Ti GPU. The hyperparameter settings of the EnADe (SA-ST-LSTM) network are shown in Table 1:

[0069] Table 1 Hyperparameter settings

[0070]

[0071] The training set of the CKIM radar echo is put into the network for training, and then the trained model is used for testing to obtain the evaluation index results:

[0072] Table 2 CKIM radar echo sequence task

[0073]

[0074]

[0075] Table 2 is the HSS and CSI data, the average data and the square error loss function of the HSS and CSI under different thresholds, the prediction results of the EnADe (SA-ST-LSTM) network are more accurate than those of other networks, it can be seen that the square loss function of the EnADe (SA-ST-LSTM) network is reduced by 6.7% compared with PredRNN, and the average HSS and CSI are increased by 2.5% and 2.9% respectively, which proves that the EnADe (SA-ST-LSTM) network unit and structure have enough potential in the radar echo sequence prediction problem. Figure 5 The prediction results of the CKIM radar echo dataset are shown, and the performance of the network is greatly improved, effectively solving the problem of different strengths and regions of radar echo sequence prediction.

[0076] Figure 6 and Figure 7The HSS and CSI frame-by-frame effects of threshold τ as 30 are shown. It can be seen from the figure that the frame-by-frame prediction results of EnADe(SA-ST-LSTM) network are generally higher than all networks, and the performance of adjacent frames is more excellent, which shows that EnADe(SA-ST-LSTM) network can not only effectively capture the mutation phenomenon of adjacent frames, but also improve the prediction results of long-term frames, so that more high-intensity areas can be predicted when performing radar echo prediction task, which proves that EnADe(SA-ST-LSTM) network effectively improves the robustness and accuracy of spatio-temporal sequence prediction.

[0077] From Figure 8 It can be seen that the loss function of the network fluctuates with different α values, and different α has a great influence on the prediction results, because α controls the updating status of long-term information and short-term information, when α = 0.9, the network is in a catastrophic forgetting state at this time, so its loss function is larger, which shows the influence of catastrophic forgetting on spatio-temporal sequence prediction, and when α = 0.1, the information updating of the unit is faster, although the effect is better than that of catastrophic forgetting, but the faster updating of short-term information will bring gradient disappearance phenomenon, when α = 0.7, the prediction effect and performance are the best, as can be seen from the figure, the performance result of the network remains within a certain interval, this fluctuation is considered to be due to the quantitative occupation of long-term memory and short-term memory in the cross-fusion process, α = 0.7 occupies 70% of the quantitative, short-term memory occupies 30% of the quantitative, at this time the effect reaches the best of the current network, the prediction effect of the network on radar echo reaches the best.

[0078] Figure 9The results of different α values are shown in the figure, including α from 0.1 to 0.9, representing the proportion of long-term memory and short-term memory information. When predicting the first two frames, all experimental results are similar, which is caused by the trend information carried by long-term memory. From the third frame, the difference between the results of different α values begins to appear. When α is 0.1 or 0.2, the network is in a catastrophic forgetting state because long-term memory state occupies too much. In this case, the network can only update under the trend information, and the information gradually shows the trend of forgetting. Therefore, from the prediction image, it can be seen that the intensity region gradually disappears, and the accurate prediction result cannot be obtained. When α is 0.8 or 0.9, the network is in a short-term memory information frequent update state. In this case, the proportion of long-term memory information trend is small, and the image updates too fast for the current time information. This situation will lead to the gradient disappearance phenomenon in the multi-step prediction process of the network. From the image, it can be seen that the network changes rapidly after the third frame, but cannot fully guarantee the long-term information, and the effect is poor. When α is 0.7, the prediction performance of the network is the best. At this time, the proportion of long-term memory and short-term memory is equivalent. The long-term memory learns the trend information from the previous network state, which is similar to the prior part of the spatio-temporal sequence unit. The short-term information is constantly updated through the trend information of the current time, so that the performance of the network reaches the best state, thereby effectively solving the saturation problem caused by catastrophic forgetting.

Claims

1. A spatio-temporal LSTM network radar echo sequence prediction method based on a self-attention mechanism, characterized in that, The method is implemented according to the following steps: Step 1, divide the CKIM radar echo data set into a training set and a test set, and preprocess to obtain a sequence image; Step 2, replace the forgetting gate mechanism in the ST-LSTM unit with a self-attention mechanism to form an SA-ST-LSTM unit; specifically: In the ST-LSTM unit, due to the over-saturation of the forgetting gate, the long-term memory and the short-term memory cannot be effectively transmitted, the forgetting gate is replaced, and the self-attention mechanism is introduced to aggregate the features, and after the important features are aggregated, the redundant information is discarded, and the attention mechanism is implemented as follows: the long-term memory state and the short-term memory H t-1 at the previous moment are used, and then the hyperparameters and are set to control the amount of information transmission, so as to realize the timely update of information, and the formula of the finally obtained SA-ST-LSTM unit is as follows: In the formula, denotes the time step, represents the number of stacked layers, denotes the short-term memory state at time layer long-term memory; represents the long-term memory at time -1, is a tunable hyperparameter; is the input state; H t-1 is the short-term memory state at time respectively represent the modulation gate, input gate and forget gate; is the spatial-temporal memory of the layer at time ; is the Sigmoid function; tanh is the tanh function; is the spatial-temporal memory of the layer at time ; is the output gate; denotes Hadamard convolution; is the self-attention mechanism; are all bias terms, are all weight matrices; Step 3, use the SA-ST-LSTM unit to build an encoding-attention-decoding network; Step 4, input the training set into the encoding-attention-decoding network for training to obtain an encoding-attention-decoding training model; Step 5, input the test set into the encoding-attention-decoding training model for testing to obtain an image prediction result and prediction data of a future frame.

2. The spatio-temporal LSTM network based on self-attention mechanism radar echo sequence prediction method according to claim 1, characterized in that, In the step 1, the training set of the CKIM radar echo dataset contains 120000 images, the test set contains 30000 images, the training set contains 24000 sequences, and the test set contains 6000 sequences; the size of all images in the two datasets is adjusted to and normalized to obtain sequence images.

3. The spatio-temporal LSTM network based on self-attention mechanism radar echo sequence prediction method according to claim 2, characterized in that, In step 3, specifically: A 3-layer CNN and a 3-layer SA-ST-LSTM unit are cross-stacked to form an encoder-decoder structure. In the horizontal direction, the long-term memory state of the SA-ST-LSTM unit... and short-term memory All transmission occurs horizontally; in the vertical direction, short-term memory states... and spacetime memory units It is passed as input to the next layer, spatiotemporal memory unit The transmission process is continuous throughout the entire network. Furthermore, an attention mechanism is added after encoding to ensure long-term memory of the encoded vector. Short-term memory and time-space memory All of them undergo an attention mechanism to further extract important information. The expression for feature extraction after encoding by the attention mechanism is shown below: wherein, represent the long-term memory state of the i-th layer, the short-term memory state of the i-th layer, and the current spatiotemporal memory state of the i-th layer, respectively represent the short-term memory state, the long-term memory state, and the current spatiotemporal memory state of the i-th layer through the attention mechanism. The 3-layer SA-ST-LSTM unit and the 3-layer CNN are stacked to form a decoding structure, and the input to the decoding structure is decoded through an attention mechanism The decoded result is finally stacked and fused through a convolution kernel to form the obtained encoding-attention-decoding network. The decoded result is finally stacked and fused through a convolution kernel to form the obtained encoding-attention-decoding network.

4. The spatio-temporal LSTM network radar echo sequence prediction method based on self-attention mechanism according to claim 3, characterized in that, In step 4, specifically: The training set is input as 5 consecutive frames as a sequence and 10 frames as a sequence of true values, and is optimized through an MSE loss function, so as to obtain a training model converged on the data set, and the prediction of the radar echo is realized through the training model, so as to judge the precipitation condition.

Images