Radar echo extrapolation method of self-attention mechanism in combination with information recall technology

Abstract

The invention discloses a radar echo extrapolation method of a self-attention mechanism in combination with an information recall technology, and the method comprises the steps: constructing a feature extraction module for a radar echo image sequence based on a space-time self-attention mechanism; introducing a time updating gate and a space updating gate to fuse the spatio-temporal characteristics, and constructing a data fusion module; constructing a data prediction module, and outputting a predicted radar echo image; according to the invention, an information recall technology and a self-attention mechanism are combined, a long-time-efficiency radar echo extrapolation scheme is provided, information loss in long-time prediction is effectively reduced, and a high-precision scheme is provided for radar echo extrapolation; the method has higher prediction precision, tiny changes in the radar echo data can be captured, and the features of the radar echo data can be better extracted and learned.

Description

Technical Field

[0001] The present invention relates to the field of radar echo sequence prediction, and in particular to a radar echo extrapolation method using a self-attention mechanism combined with information recall technology. Background Art

[0002] As a type of severe convective weather, short-term heavy rainfall is characterized by short duration, strong localization, and high variability in factors. It can easily cause localized flooding, mountain disasters, and disruptions to telecommunications and transportation. Short-term forecasts primarily use weather radar echo data.

[0003] Currently, the most important tool for short-term precipitation forecasting is radar echo extrapolation. Its goal is to leverage historical observational data to accurately and timely predict changes in radar echo reflectivity over a local area over the next two hours or more. There are two main approaches to radar echo extrapolation. Traditional methods include centroid tracking, cross-correlation, and optical flow. These methods provide good predictions of cloud translation, but struggle to predict the formation and dissipation of edge precipitation clouds. Deep learning methods, which leverage historical data, have effectively addressed this issue. These methods include ConvLSTM, PredRNN, and GAN. However, these methods suffer from information loss, inaccurate predictions of radar echo trajectory, and poor generalization. Summary of the Invention

[0004] The purpose of the present invention is to provide a radar echo extrapolation method that combines information recall technology with a self-attention mechanism. This method combines information recall technology and a self-attention mechanism to effectively reduce information loss in long-term predictions and provide a high-precision solution for radar echo extrapolation.

[0005] To achieve the above functions, the present invention designs a radar echo extrapolation method that combines a self-attention mechanism with information recall technology, and executes the following steps S1 to S4 to construct and apply a radar echo extrapolation model:

[0006] Step S1: Based on the ST-LSTM neural network (spatial-temporal long short-term memory network), a feature extraction module is constructed. The radar echo image sequence is used as input, and the hidden state and memory state are dynamically updated using the spatiotemporal self-attention gating mechanism, and a new hidden state is output at each time step.

[0007] Step S2: Construct a data fusion module, take the hidden state obtained in step S1 as input, introduce a time update gate and a space update gate to fuse the spatiotemporal features, generate new time features and new spatial features, and obtain the hidden state;

[0008] Step S3: construct a data prediction module, take the hidden state obtained in step S2 as input, use time-step prediction, and output the predicted radar echo image;

[0009] Step S4: train the radar echo extrapolation model constructed in steps S1 to S3 to obtain a trained radar echo extrapolation model, deploy the trained radar echo extrapolation model, and complete the radar echo extrapolation task.

[0010] As a preferred technical solution of the present invention: Step S1 specifically comprises the following steps:

[0011] Step S1.1: For the input radar echo image sequence, extract the input frame through convolution operation Hidden State Time characteristics Spatial characteristics

[0012] Step S1.2: Use the gating mechanism of spatiotemporal self-attention to dynamically update the hidden state and memory state, and output the new hidden state at each time step, as shown in the following formula:

[0013]

[0014] Among them, W i 、W f 、W o 、W g 、W i ′、W f ′ 、W g 、U i 、U f 、U o 、U g 、U i ′ 、U f ′ 、U o , Q O 、U g Represents various parameters during model training, σ() represents the Sigmoid function, tanh represents the activation function, and · represents the convolution calculation; and Represent the hidden state of the lth layer at time t-1 and the hidden state of the l-1th layer at time t respectively; Represents the time memory information of the lth layer at time t-1, Represents the spatial memory information of the l-1th layer at time t; i t , f t , o t , g tRepresent the input gate, forget gate, output gate and input candidate modulation gate respectively, b i 、b f 、b o 、b g Represent the bias of the input gate, forget gate, output gate and input candidate modulation gate respectively; i t ′ is the input gate of spatial information, f t ′The forget gate of spatial information, o t ′ is the output gate of spatial information, b i ′、b f ′、b o ′、b g ′ respectively represent the bias when the input gate, forget gate, output gate and input candidate modulation gate control the spatial features.

[0015] As a preferred technical solution of the present invention: input frame in spatiotemporal self-attention mechanism Hidden State Time characteristics Spatial characteristics The format is (batch size, number of input channels, height, width).

[0016] As a preferred technical solution of the present invention: the data fusion module in step S2 is as follows:

[0017]

[0018] S t =Softmax(Q s ×(K s ) T )

[0019]

[0020] T l =Softmax(Q t ×(K t ) T )

[0021] G s =σ(W gs ·S t )

[0022] G t =σ(W gt ·T l )

[0023]

[0024] Among them, V s , Q s, K s Expression The memory information related to space obtained by self-attention calculation is Q s , K s After matrix multiplication and Softmax calculation, normalization is performed to obtain S t ; is the hidden state obtained after self-attention calculation; Q t , K t Q is the time-related memory information obtained by self-attention calculation. t , K t Perform matrix multiplication and Softmax calculation to get T l ; To control the ratio of spatiotemporal information fusion, two update gates are designed, where G t G is the time update gate, which aggregates the time memory information; s is the spatial update gate, which aggregates the spatial memory information; W gs 、W gt 、W tt 、W ts 、W ss 、W st Represents various parameters during model training.

[0025] As a preferred technical solution of the present invention: the data prediction module described in step S3 is composed of a batch normalization layer, a Dropout layer, a fully connected layer, and an output interface.

[0026] As a preferred technical solution of the present invention: the output of the data prediction module is a list of multiple time steps, and each element in the list is a predicted radar echo image tensor at a future time point.

[0027] As a preferred technical solution of the present invention: the radar echo extrapolation model adopts time-step prediction. For input time step t less than the input duration, the real frame radar echo image is directly used as input; for output time step t greater than or equal to the input duration, whether to use the real frame or the generated frame is decided based on the mask, and the convolution layer is used to convert the output of the ST-LSTM neural network into a predicted frame for the next time step.

[0028] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0029] The radar echo extrapolation model designed in this paper has a simple framework and advanced technology. It combines the self-attention mechanism with the ST-LSTM neural network framework, namely (RAST-LSTM). This invention incorporates an information recall mechanism to address the problem of information loss when processing long-term memories in existing models, achieving better heavy rainfall warning effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a structural block diagram of a radar echo extrapolation method using a self-attention mechanism combined with information recall technology provided by an embodiment of the present invention;

[0031] Figure 2 It is a visualization diagram of the prediction results of each model under a typical sequence according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0033] The radar echo extrapolation method of the self-attention mechanism combined with the information recall technology provided in the embodiment of the present invention performs the following steps S1 to S4, referring to Figure 1 , build and apply the radar echo extrapolation model:

[0034] Step S1: Based on the ST-LSTM neural network, a feature extraction module is constructed. The radar echo image sequence is used as input, the hidden state and memory state are dynamically updated using the gating mechanism, and the new hidden state is output at each time step.

[0035] The specific steps of step S1 are as follows:

[0036] Step S1.1: For the input radar echo image sequence, extract the input frame through convolution operation Hidden State Time characteristics Spatial characteristics

[0037] Input frames in spatiotemporal self-attention mechanism Hidden State Time characteristics Spatial characteristics The format is (batch size, number of input channels, height, width), that is, (batch_size,in_channel,height,width).

[0038] Step S1.2: Utilize the gating mechanism of spatiotemporal self-attention to dynamically update the hidden state and memory state, and output a new hidden state at each time step. Global dependency modeling is used to capture the correlation between time steps through the self-attention mechanism. The self-attention mechanism is used to generate temporal and spatial feature information, and residual connections are used to combine input and context feature information. Radar data operations are distributed to each time step, input radar data is processed frame by frame, and convolution projections are performed on the keys and values ​​of the time series. The self-attention mechanism is used to calculate the correlation between time steps, and the weighted sum is used to generate context features. The feature extraction module is specifically as follows:

[0039]

[0040] Among them, W i 、W f 、W o 、W g 、W i ′、W f ′ 、W g 、U i 、U f 、U o 、U g 、U i ′ 、U f ′ 、U o , Q O 、U g Represents various parameters during model training, σ() represents the Sigmoid function, tanh represents the activation function, and · represents the convolution calculation; and Represent the hidden state of the lth layer at time t-1 and the hidden state of the l-1th layer at time t respectively; Represents the time memory information of the lth layer at time t-1, represents the spatial memory information of the l-1th layer at time t, Represents the spatial memory information of the lth layer at time t; i t , f t , o t , g t Represent the input gate, forget gate, output gate and input candidate modulation gate respectively, b i 、b f 、b o 、b g Represent the bias of the input gate, forget gate, output gate and input candidate modulation gate respectively. t ′ is the input gate of spatial information, which controls the influence of the spatial features of the current time step on the memory. t ′The forget gate of spatial information controls the spatial memory of the previous time step o t ′ is the output gate of spatial information, which controls the influence of spatial features on the output. g t ′ is the input candidate modulation gate of spatial information. b i ′、b f ′、b o ′、b g ′ respectively represent the bias when the input gate, forget gate, output gate and input candidate modulation gate control the spatial features.

[0041] Step S2: Construct a data fusion module, take the hidden state obtained in step S1 as input, introduce a time update gate and a space update gate to fuse the spatiotemporal features, generate new time features and new spatial features, and obtain the hidden state;

[0042] The data fusion module described in step S2 is as follows:

[0043]

[0044] S t =Softmax(Q s ×(K s ) T )

[0045]

[0046] T l =Softmax(Q t ×(K t ) T )

[0047] G s =σ(W gs ·S t )

[0048] G t =σ(W gt ·T l )

[0049]

[0050] Among them, V s , Q s , K s Expression The memory information related to space obtained by self-attention calculation is Q s , K s After matrix multiplication and Softmax calculation, normalization is performed to obtain S t . is the hidden state obtained after self-attention calculation; Q t, K t is the time-related memory information obtained through self-attention calculation. Q t , K t are multiplied in matrix and Softmax calculation is performed to obtain T l ; To control the proportion of spatio-temporal information fusion, two update gates are designed. Among them, G t is the time update gate, which aggregates the time memory information. G s is the space update gate, which aggregates the space memory information. W gs , W gt , W tt , W ts , W ss , W st represent various parameters during the training of this model.

[0051] Step S3: Construct a data prediction module. Taking the hidden state obtained in step S2 as the input, adopt step-by-step time prediction to output the predicted radar echo image; the data prediction module consists of a batch normalization layer, a Dropout layer, a fully connected layer, and an output interface. The output of the data prediction module is a list of multiple time steps, and each element in the list is a predicted radar echo image tensor at a future time point. In one embodiment, the length of the output list is n_steps, and the default value is 6.

[0052] Step S4: Train the radar echo extrapolation model constructed in steps S1 - S3 to obtain a trained radar echo extrapolation model, deploy the trained radar echo extrapolation model, and complete the radar echo extrapolation task.

[0053] The radar echo extrapolation model adopts step-by-step time prediction. For the input time step t less than the input duration (i.e., t < input_length), directly use the real-frame radar echo image as the input; for the output time step t greater than or equal to the input duration (i.e., t >= input_length), based on the mask, decide whether to use the real frame or the generated frame, and use the convolutional (conv_last) layer to convert the output of the ST-LSTM neural network into the predicted frame for the next time step. The conv_last layer is the "output mapping layer" inside the ST-LSTM neural network, which is responsible for projecting the internal state (memory) of the ST-LSTM neural network into the output frame features for subsequent prediction, decoding, or reconstruction.

[0054] In one embodiment, the dataset input to the radar echo extrapolation model consists of 14,000 processed radar echo image sequences, divided into training, validation, and test sets. The training set contains 8,000 sequences, the validation set contains 2,000 sequences, and the test set contains 4,000 sequences. Each sequence consists of 15 radar reflectivity images, with the first five serving as input data and the last ten serving as the desired output.

[0055] The following is an application embodiment of the present invention:

[0056] The present invention combines the self-attention mechanism and the ST-LSTM neural network framework, namely (RAST-LSTM), and incorporates the information recall mechanism to solve the problem of information loss in the existing model when processing long-term memory, so as to achieve a better heavy rainfall warning effect. Finally, the three indicators commonly used in the meteorological industry, CSI (Critical Success Index, the larger the better), FAR (False Alarm Rate, the smaller the better), and HSS (Heidke Skill Score, the larger the better) are used as evaluation indicators to analyze the prediction accuracy of various models. At the same time, experiments were conducted on the collected Guangzhou station radar echo data set to compare the prediction results of ConvGRU, ConvLSTM, PredRNN, and PredRNN++ networks. The experimental results are shown in Table 1, Table 2, and Table 3. Figure 2 As shown:

[0057] Table 1: CSI scores of all models at Guangzhou station

[0058] Model τ=20 τ=30 τ=40 ConvGRU 0.55 0.41 0.11 ConvLSTM 0.49 0.37 0.21 PredRNN 0.64 0.52 0.27 PredRNN++ 0.66 0.54 0.28 RAST-LSTM 0.66 0.55 0.29

[0059] Table 1 shows the CSI scores of all models at the Guangzhou station. It can be seen from the table that the RAST-LSTM proposed in this paper performs best. Compared with the PredRNN model, RAST-LSTM improves by 2%, 3%, and 2% on 20dz, 30dz, and 40dz, respectively.

[0060] Table 2 FAR scores of all models at Guangzhou station

[0061] Model τ=20 τ=30 τ=40 ConvGRU 0.39 0.49 0.66 ConvLSTM 0.50 0.59 0.62 PredRNN 0.15 0.21 0.35 PredRNN++ 0.15 0.20 0.34 RAST-LSTM 0.17 0.18 0.22

[0062] Table 2 shows the FAR scores of all models at the Guangzhou station. The table shows that the RAST-LSTM model proposed in this paper performs best, reducing the false alarm rate by 2%, 3%, and 13% at different thresholds compared to the baseline model. In high-intensity echoes, RAST-LSTM significantly improves the false alarm rate. Compared to the second-best PredRNN++ model, the RAST-LSTM model reduces the false alarm rate by 2%, 2%, and 12% at different thresholds.

[0063] Table 3 HSS scores of all models at Guangzhou station

[0064] Model τ=20 τ=30 τ=40 ConvGRU 0.66 0.51 0.18 ConvLSTM 0.53 0.45 0.32 PredRNN 0.73 0.65 0.41 PredRNN++ 0.74 0.66 0.42 RAST-LSTM 0.75 0.64 0.42

[0065] Table 3 shows the HSS scores of all models at the Guangzhou station. It can be seen from the table that the RAST-LSTM proposed in this paper performs best, with improvements of 2% and 1% at 20dz and 40dz respectively compared with the baseline model.

[0066] The content that tabular data can display is limited and cannot reflect the specific performance of the model in the actual task. Therefore, Figure 2 This figure shows the visualization results of each of the aforementioned models for a typical sequence using Guangzhou data. The color scale in the lower right corner represents radar echo intensity levels from 10 to 70+ dBZ, with a step of 5 dBZ. The first row of the figure shows the ground truth, i.e., the actual radar echo map. The five rows below the dashed line, totaling 50 images, show the visualization prediction results of each model for the given typical sequence.

[0067] Figure 2 The first row of the group sequence contains a large number of medium- and high-intensity echoes, effectively demonstrating the model's predictive ability for medium- and high-intensity echoes. Furthermore, the presence of a range of medium- and low-intensity echoes surrounding the sequence indicates the model's ability to learn the overall shape and trend of the echoes. Specifically, for each model, the ConvGRU model accurately predicts echoes within 30 minutes. However, after 30 minutes, medium- and high-intensity echoes exhibit significant errors and are too low in intensity, indicating that the ConvGRU model's ability to learn high-intensity echoes over longer prediction times is weak. For the ConvLSTM, the difference between the high-intensity area and the ground-truth area increases over time, indicating that the ConvLSTM has difficulty effectively learning the motion trends and intensity changes of radar echoes, making it difficult to use in actual predictions.

[0068] Figure 2The fourth row shows the visualization results of the PredRNN model. Low-intensity echoes are well preserved, with positions and shapes similar to the true values. Although the values ​​of high-intensity echoes are somewhat reduced, this reflects its ability to learn echo motion trends. The PredRNN++ model, on the other hand, has an overall predicted shape similar to the true value, but the prediction deviation in high-intensity areas is large, and most low- and medium-intensity areas are retained, indicating that it still suffers from the vanishing gradient problem.

[0069] The visualization results of the proposed RAST-LSTM model show that, thanks to its use of spatiotemporal representations transferred from attention units and information recall mechanisms, the model excels in predicting echo motion trends in different partitions. Specifically, in the longer-term prediction results, the high-intensity region of the true value is concentrated in the lower right center. The high-intensity echo region and size predicted by the model generally match the true value, indicating that the model has higher prediction accuracy for local details and is able to capture subtle changes in radar echo data. This demonstrates that the present invention is capable of better extracting and learning the characteristics of radar echo data.

[0070] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in this field without departing from the spirit of the present invention.

Claims

1. A radar echo extrapolation method based on a self-attention mechanism combined with information recall technology is characterized by: Perform the following steps S1 to S4 to construct and apply the radar echo extrapolation model: Step S1: Based on the ST-LSTM neural network, a feature extraction module is constructed. The radar echo image sequence is used as input, and the hidden state and memory state are dynamically updated using the spatiotemporal self-attention gating mechanism, and the new hidden state is output at each time step. Step S2: Construct a data fusion module, take the hidden state obtained in step S1 as input, introduce a time update gate and a space update gate to fuse the spatiotemporal features, generate new time features and new spatial features, and obtain the hidden state; Step S3: construct a data prediction module, take the hidden state obtained in step S2 as input, use time-step prediction, and output the predicted radar echo image; Step S4: train the radar echo extrapolation model constructed in steps S1 to S3 to obtain a trained radar echo extrapolation model, deploy the trained radar echo extrapolation model, and complete the radar echo extrapolation task.

2. The radar echo extrapolation method using a self-attention mechanism combined with information recall technology according to claim 1, characterized in that: The specific steps of step S1 are as follows: Step S1.1: For the input radar echo image sequence, extract the input frame through convolution operation Hidden State Time characteristics Spatial characteristics Step S1.2: Use the gating mechanism of spatiotemporal self-attention to dynamically update the hidden state and memory state, and output the new hidden state at each time step, as shown in the following formula: Among them, W i 、W f 、W o 、W g 、W i ′、W f ′、W g 、U i 、U f 、U o 、U g , U′ i , U′ f 、U o , Q O 、U g Represents various parameters during model training, σ() represents the Sigmoid function, tanh represents the activation function, and · represents the convolution calculation; and Represent the hidden state of the lth layer at time t-1 and the hidden state of the l-1th layer at time t respectively; Represents the time memory information of the lth layer at time t-1, Represents the spatial memory information of the l-1th layer at time t; i t , f t , o t , g t Represent the input gate, forget gate, output gate and input candidate modulation gate respectively, b i 、b f 、b o 、b g Represent the bias of the input gate, forget gate, output gate and input candidate modulation gate respectively; i t ′ is the input gate of spatial information, f t ′The forget gate of spatial information, o t ′ is the output gate of spatial information, b i ′、b f ′、b o ′、b g ′ respectively represent the bias when the input gate, forget gate, output gate and input candidate modulation gate control the spatial features.

3. The radar echo extrapolation method using a self-attention mechanism combined with information recall technology according to claim 2, characterized in that: Input frames in spatiotemporal self-attention mechanism Hidden State Time characteristics Spatial characteristics The format is (batch size, number of input channels, height, width).

4. The radar echo extrapolation method using a self-attention mechanism combined with information recall technology according to claim 1, characterized in that: The data fusion module described in step S2 is as follows: S t =Softmax(Q s ×(K s ) T ) T l =Softmax(Q t ×(K t ) T ) G s =σ(W gs ·S t ) G t =σ(W gt ·T l ) Among them, V s , Q s , K s Expression The memory information related to space obtained by self-attention calculation is Q s , K s After matrix multiplication and Softmax calculation, normalization is performed to obtain S t ; is the hidden state obtained after self-attention calculation; Q t , K t Q is the time-related memory information obtained by self-attention calculation. t , K t Perform matrix multiplication and Softmax calculation to get T l ; To control the ratio of spatiotemporal information fusion, two update gates are designed, where G t G is the time update gate, which aggregates the time memory information; s is the spatial update gate, which aggregates the spatial memory information; W gs 、W gt 、W tt 、W ts 、W ss 、W st Represents various parameters during model training.

5. The radar echo extrapolation method using a self-attention mechanism combined with information recall technology according to claim 1, characterized in that: The data prediction module described in step S3 consists of a batch normalization layer, a Dropout layer, a fully connected layer, and an output interface.

6. The radar echo extrapolation method using a self-attention mechanism combined with information recall technology according to claim 1, characterized in that: The output of the data prediction module is a list of multiple time steps, where each element in the list is a predicted radar echo image tensor at a future time point.

7. The radar echo extrapolation method using a self-attention mechanism combined with information recall technology according to claim 1, characterized in that: The radar echo extrapolation model adopts time-step prediction. For input time step t less than the input duration, the real frame radar echo image is directly used as input. For output time step t greater than or equal to the input duration, whether to use the real frame or the generated frame is decided based on the mask, and the convolutional layer is used to convert the output of the ST-LSTM neural network into the predicted frame of the next time step.

Images