Radar echo sequence prediction method based on transform-gated spatiotemporal long short-term memory unit

Abstract

The application discloses a radar echo sequence prediction method based on a transform gate space-time long short-term memory unit, and specifically comprises the following steps: dividing a data set into a training set and a test set; then, pre-processing images in the data set to obtain a radar echo sequence; on the basis of a space-time long short-term memory unit, a transform gate mechanism is used to improve a forget gate, so as to construct a TG-ST-LSTM structure unit in a network structure; a TG-PredRNN network structure is obtained in a TG-ST-LSTM structure unit stacking mode, and the network structure is trained by using training set data; test set data is input into the network for testing, and then the obtained images are restored, so that a prediction result image is finally obtained. Compared with a classic ST-LSTM and a ConvLSTM structure unit, the structure unit with the transform gate mechanism can significantly improve the ability to capture short-term mutation phenomena and space-time correlation of the radar echo sequence.

Description

Technical Field

[0001] This invention belongs to the field of radar echo sequence prediction technology, specifically relating to a radar echo sequence prediction method based on transform gate spatiotemporal long short-term memory units. Background Technology

[0002] Weather affects every aspect of people's lives, from agricultural production to natural disasters, from travel to production and services. Therefore, weather observation and forecasting have been a concern and a problem to be solved throughout history. Knowing future weather conditions in advance and taking effective countermeasures is essential, and using deep learning to predict the changing trends of radar echo sequences to determine precipitation has become a research hotspot in the field of smart meteorology. This nowcasting of precipitation is essentially a spatiotemporal sequence prediction problem that takes past radar echo sequences as input and outputs a fixed number (usually greater than one) of future radar echo sequences. However, due to the non-stationarity of spatiotemporal sequences, especially when multi-step prediction is necessary, unless the prediction model can accurately capture the spatiotemporal structure of the data, the predicted radar echo sequences will be blurred and noisy, leading to reduced prediction accuracy. Therefore, capturing spatiotemporal correlations is particularly important. Summary of the Invention

[0003] The purpose of this invention is to provide a radar echo sequence prediction method based on transform gate spatiotemporal long short-term memory units, thereby improving the spatiotemporal correlation of spatiotemporal sequence prediction and the performance of radar echo sequence prediction.

[0004] The technical solution adopted in this invention is a radar echo sequence prediction method based on transform gate spatiotemporal long short-term memory units, which is implemented according to the following steps:

[0005] Step 1: Use the MovingMNIST dataset and the Standardized RAdar Dataset 2020 radar echo dataset, and divide the datasets into training and testing sets; then preprocess the images in the two datasets to obtain radar echo sequences that can be input into the network structure.

[0006] Step 2: Based on the spatiotemporal long short-term memory unit, the forget gate is improved by using the transformation gate mechanism, thereby constructing the TG-ST-LSTM structural unit in the network structure.

[0007] Step 3: Obtain the TG-PredRNN network structure by stacking TG-ST-LSTM structural units;

[0008] Step 4: Train the TG-PredRNN network structure using the training set data;

[0009] Step 5: Input the test set data into the TG-PredRNN network for testing, then restore the obtained image to finally obtain the predicted result image.

[0010] The invention is further characterized in that,

[0011] In step 1, the specific steps are as follows:

[0012] The MovingMNIST dataset was divided into a training set and a test set, with the training set containing 20,000 sequences and the test set containing 5,000 sequences. In the SRAD2020 radar echo dataset, 17,031 consecutive images were used as the training set and 3,969 consecutive images were used as the test set. Then, the size of all images in both datasets was adjusted to 64×64×1, grayscale was performed, and the images were sliced ​​to a 16×16×16 format to reduce the computational cost.

[0013] In step 2, the specific steps are as follows:

[0014] In the ST-LSTM cell structure, the forget gate f is passed through the 1-tanh function. t The saturation range of the Sigmoid function is compressed, making the forget gate f t The output is mapped to the range [0, 0.75] through a transformation gate. After processing by the transformation gate, values ​​close to 1 are reduced to 0.25, values ​​close to 0 are made closer to 1, and values ​​near the middle are compressed to 0.5, thus obtaining the TG-ST-LSTM structural unit, whose expression is as follows:

[0015]

[0016]

[0017]

[0018] tr = 1 - tanh(f) t )

[0019]

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026] In the formula, g t Indicates the input modulation gate; i t Indicates the input gate; f t Represents the forget gate; g′ t , i′ t f′ t These represent the modulation gate, input gate, and forget gate in another set of gate mechanisms, respectively; tr represents the transform gate mechanism; This represents the l-th layer of long-term memory at time t; This represents the (l-1)th layer of long-term memory at time t; This represents the l-th layer of spatiotemporal memory at time t; Represents the spatiotemporal memory of layer l-1 at time t; o t Indicates the output gate; denoted by t, representing the short-term memory state of the l-th layer at time t; tanh represents the tanh function; σ represents the sigmoid function, W xg W xi W xf W hg W hi W hf W mg W mi W mf W 1×1×l , W′ xg , W′ xi , W′ xf W xo W ho W co W mo Both represent weights; b g b i b f , b′ g , b′ i b′ f b o Both indicate bias.

[0027] Step 3, the specific steps are as follows:

[0028] The TG-PredRNN network structure consists of four stacked TG-ST-LSTM structural units; specifically, in the horizontal direction, it transmits the long-term memory state C from the previous time step. t and hidden state H t ;Utilize the hidden state H from the previous time step t-1 and long-term memory state C t-1 The hidden state H at the current moment t and long-term memory state Ct To enable horizontal interaction and ensure the updating of each hidden state and long-term memory state, the initial hidden state H... t and memory state C t Set to zero; in the vertical direction, pass the hidden state Ht and the spacetime memory unit M. t H will be hidden t and spacetime memory unit M t The hidden states H obtained in the last layer are connected together and passed as input to form the TG-PredRNN network structure. t The features are fused with the memory unit Mt to obtain the final prediction result.

[0029] Step 4 involves the following steps:

[0030] The training set images from the MovingMNIST dataset are input into the TG-PredRNN network structure every ten frames. The output results are then optimized using the mean squared error loss function. Image similarity, peak signal-to-noise ratio, and image perceptual similarity are selected as evaluation metrics to evaluate the TG-PredRNN network structure.

[0031] The training images from the SRAD2020 dataset were input into the TG-PredRNN network structure every ten frames. The output results were optimized using the mean squared error loss function. Different thresholds τ were selected to evaluate the amount of rainfall, with τ values ​​of 20 and 30 corresponding to different amounts of rainfall. The critical success index, detection probability, and Heidke skill score were used as evaluation metrics to evaluate the TG-PredRNN network structure.

[0032] The beneficial effects of this invention are that it addresses the traditional radar echo sequence prediction task by proposing a TG-ST-LSTM structural unit and its corresponding TG-PredRNN network structure. Compared with the traditional ST-LSTM structural unit, this structural unit with a transform gate mechanism can compress the amount of information in the image to an effective range of variation, avoiding to some extent the oversaturation interval of the Sigmoid activation function. Compared with the classic ST-LSTM and ConvLSTM structural units, the structural unit with a transform gate mechanism can significantly improve the ability to capture short-term abrupt changes and spatiotemporal correlations in radar echo sequences. Attached Figure Description

[0033] Figure 1 This is a diagram of the TG-ConvLSTM structure unit in the radar echo sequence prediction method based on transform gate spatiotemporal long short-term memory unit of this invention;

[0034] Figure 2This is a diagram of the ST-LSTM structure unit in the radar echo sequence prediction method based on transform gate spatiotemporal long short-term memory unit of this invention;

[0035] Figure 3 This is a diagram of the TG-ST-LSTM structure unit in the radar echo sequence prediction method based on transform gate spatiotemporal long short-term memory unit of this invention;

[0036] Figure 4 This is a diagram of the TG-PredRNN network structure in the radar echo sequence prediction method based on transform gate spatiotemporal long short-term memory units in this invention.

[0037] Figure 5 The training flowchart of the TG-PredRNN network structure in the radar echo sequence prediction method based on transform gate spatiotemporal long short-term memory unit in this invention;

[0038] Figure 6 The test flowchart of the TG-PredRNN network structure in the radar echo sequence prediction method based on transform gate spatiotemporal long short-term memory unit in this invention;

[0039] Figure 7 This is the SSIM evaluation graph of the prediction results of ten frames on the MovingMNIST dataset in this embodiment of the invention;

[0040] Figure 8 This is the LPIPS evaluation graph of the prediction results of ten frames on the MovingMNIST dataset in this embodiment of the invention.

[0041] Figure 9 This is a PSNR evaluation graph of ten frames of prediction results on the MovingMNIST dataset in an embodiment of the present invention.

[0042] Figure 10 This is a ten-frame predicted POD result diagram of the SRAD2020 radar echo dataset when τ=20 in this embodiment of the invention;

[0043] Figure 11 This is a diagram showing the predicted HSS results of ten frames of the SRAD2020 radar echo dataset when τ=20 in this embodiment of the invention.

[0044] Figure 12 This is a diagram showing the predicted CSI results of ten frames of the SRAD2020 radar echo dataset when τ=20 in this embodiment of the invention.

[0045] Figure 13 This is a ten-frame predicted POD result diagram of the SRAD2020 radar echo dataset when τ=30 in this embodiment of the invention;

[0046] Figure 14This is a diagram showing the predicted HSS results of ten frames of the SRAD2020 radar echo dataset when τ=30 in this embodiment of the invention.

[0047] Figure 15 This is a diagram showing the predicted CSI results of ten frames of the SRAD2020 radar echo dataset when τ=30 in this embodiment of the invention.

[0048] Figure 16 This is a sequence prediction result diagram of the MovingMNIST dataset in an embodiment of the present invention;

[0049] Figure 17 This is a sequence prediction result diagram of the SRAD2020 radar echo dataset in an embodiment of the present invention. Detailed Implementation

[0050] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0051] This invention relates to a radar echo sequence prediction method based on transform gate spatiotemporal long short-term memory units, which is implemented according to the following steps:

[0052] Step 1: Using the classic spatiotemporal sequence datasets MovingMNIST and Standardized RAdarDataset 2020 (SRAD2020), preprocess the sequence images in the datasets to obtain radar echo sequences that can be input into the network structure.

[0053] The specific approach is as follows: The MovingMNIST dataset is divided into a training set and a test set, with the training set containing 20,000 sequences and the test set containing 5,000 sequences; in the SRAD2020 radar echo dataset, 17,031 consecutive images are used as the training set and 3,969 consecutive images are used as the test set; then, the size of all images in both datasets is adjusted to 64×64×1, grayscale is performed, and the images are sliced ​​to make them 16×16×16 format to reduce the computational cost;

[0054] Step 2: Based on the Spatiotemporal Long Short-Term Memory (ST-LSTM) unit, the forgetting gate is improved by using the transformation gate mechanism, thereby constructing the TG-ST-LSTM structural unit in the network structure. This enhances the adaptability of the network structure to short-term mutations and long-term memory, and strengthens the network structure's ability to model short-term spatiotemporal dynamic characteristics.

[0055] To address the spatial data redundancy problem in FC-LSTM structures, the ConvLSTM structure extends the FC-LSTM structure by converting the multiplication vectors in the latter into convolutional structures, thus mitigating the spatial data redundancy issue. Subsequently, the TG-ConvLSTM structure was proposed based on the traditional ConvLSTM structure. The TG-ConvLSTM unit is shown below. Figure 1 As shown, this unit mainly improves the forget gate mechanism of the ConvLSTM structure unit, further compressing the data stream so that more data streams are compressed into the range that the forget gate can capture. This method is more conducive to capturing the dependencies between data.

[0056] The unit structure of ST-LSTM is as follows: Figure 2 As shown, the ST-LSTM structure unit is based on the ConvLSTM structure unit with the addition of a spatiotemporal memory unit module M. t The original memory state C t and the new spatiotemporal memory state M t Combining these elements, a new recurrent network unit is derived. This unit utilizes the four gate structures from LSTM and adds a spatiotemporal memory state M to the LSTM structure unit. t It consists of another set of input gates i′ t Input modulation gate g′ t Forgotten Gate f′ t and output gate o′ t The final hidden state Ht of each node depends on a nonlinear combination of two memory states in the horizontal and zigzag directions. ST-LSTM structural units are connected in the channel dimension of the memory state three-dimensional tensor. t and M t Then, a 1 × 1 convolutional layer is applied and its dimensionality is reduced using the tanh activation function, so that the hidden state H t It has the same dimensions as the memory state, and is then controlled by an output gate from signals originating in two directions. t Further seamlessly integrate dual memory states.

[0057] The specific method is as follows: In the cell structure of ST-LSTM, the forget gate f is passed through the 1-tanh function. t The saturation range of the Sigmoid function is compressed, making the forget gate f t The output is mapped to the range [0, 0.75] through a transformation gate. After processing by the transformation gate, values ​​close to 1 are reduced to 0.25, values ​​close to 0 are made closer to 1, and values ​​near the middle are compressed to 0.5, thus obtaining the TG-ST-LSTM structural unit, as shown below. Figure 3As shown, its expression is as follows:

[0058]

[0059]

[0060]

[0061] tr = 1 - tanh(f) t )

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069] In the formula, g t Indicates the input modulation gate; i t Indicates the input gate; f t Represents the forget gate; g′ t , i′ t f′ t These represent the modulation gate, input gate, and forget gate in another set of gate mechanisms, respectively; tr represents the transform gate mechanism; This represents the l-th layer of long-term memory at time t; This represents the (l-1)th layer of long-term memory at time t; This represents the l-th layer of spatiotemporal memory at time t; Represents the spatiotemporal memory of layer l-1 at time t; o t Indicates the output gate; denoted by t, representing the short-term memory state of the l-th layer at time t; tanh represents the tanh function; σ represents the sigmoid function, W xg W xi W xf W hg W hi W hf W mg W mi W mf W 1×1×1 , W′ xg , W′ xi , W′xf W xo W ho W co W mo Both represent weights; b g b i b f b′ g b′ i b′ f b o All represent bias; l represents the number of stacked cell layers; t represents the time step; ⊙ represents Hadamard convolution; * represents convolution operation;

[0070] This will forget the gate f t The practice of compressing the range of transmitted data values ​​to intervals within the significant variation range of the Sigmoid function helps capture dependencies between data and significantly reduces the amount of insignificant information in the data stream within the forget gate saturation region. This approach enhances the dependencies between short-term information while preserving long-term dependencies. Compared to ST-LSTM structural units, this method of compressing the data stream through the transform gate to the most significant variation range avoids, to some extent, the oversaturation interval of the Sigmoid activation function. This reduces the problem of insignificant changes in short-term effective information within the saturation region, enhancing the ability of TG-ST-LSTM structural units to capture short-term mutations and spatiotemporal correlations in spatiotemporal sequence data.

[0071] Step 3: Obtain the TG-PredRNN network structure by stacking TG-ST-LSTM structural units;

[0072] The specific implementation is as follows: The TG-PredRNN network structure is as follows: Figure 4 As shown, a spatiotemporal memory unit is incorporated to enable interaction between layers, and it is composed of four stacked TG-ST-LSTM structural units. Horizontally, it transmits the long-term memory state C from the previous time step. t and hidden state H t Utilizing the hidden state H from the previous time step t-1 and long-term memory state C t-1 The hidden state H at the current moment t and long-term memory state C t To enable horizontal interaction and ensure the updating of each hidden state and long-term memory state, the initial hidden state H... t and memory state C t Set to zero. Pass the hidden state H in the vertical direction. t and spacetime memory unit M t H will be hidden t and spacetime memory unit M tThe hidden states H obtained in the last layer are connected together and passed as input to form the TG-PredRNN network structure. t and memory unit M t Feature fusion is performed to obtain the final prediction result;

[0073] Step 4: Train the TG-PredRNN network structure using the training set data;

[0074] The specific approach is as follows: the training set images from the MovingMNIST dataset are input into the TG-PredRNN network structure every ten frames, and the output results are optimized by the mean squared error (MSE) loss function. The image similarity (SSIM), peak signal-to-noise ratio (PSNR), and perceptual image similarity (LPIPS) are selected as evaluation metrics to evaluate the TG-PredRNN network structure.

[0075] The training images from the SRAD2020 dataset were input into the TG-PredRNN network structure every ten frames. The output results were optimized using the mean squared error (MSE) loss function. Different thresholds τ were selected to evaluate the amount of rainfall, with τ values ​​of 20 and 30 corresponding to different amounts of rainfall. The Critical Success Index (CSI), Probability of Detection (POD), and Heidke Skill Score (HSS) were used as evaluation metrics to evaluate the TG-PredRNN network structure.

[0076] Comparing the experimental results from different network structures clearly shows the impact of the forget gate on the overall outcome. Experiments demonstrate that improving the forget gate and constraining its variation range yields excellent results. The training process is as follows: Figure 5 As shown.

[0077] Step 5: Input the test set data into the TG-PredRNN network for testing, then restore the obtained 16×16×16 image to finally obtain a 64×64×1 prediction result image; the testing process is as follows. Figure 6 As shown.

[0078] Example

[0079] In the MovingMNIST general dataset experiments, the training set used 20,000 sequences, and the test set used 5,000 sequences. The hyperparameter settings involved in the experiments are shown in Table 1.

[0080] Table 1 Hyperparameter settings for the MovingMNIST dataset

[0081]

[0082]

[0083] In the process of image evaluation, SSIM, PSNR and LPIPS were used as evaluation indicators, and the results are shown in Table 2.

[0084] Table 2 Experimental results on the MovingMNIST dataset

[0085]

[0086] As shown in Table 2, it can be seen that in the evaluation of network structure, the TG-PredRNN network structure has higher evaluation indicators than other models. Figure 7 , Figure 8 and Figure 9 The image shows frame-by-frame evaluation graphs for SSIM, LPIPS, and PNSR on this dataset. It can be seen that the evaluation metrics of all models decrease with increasing frame count, while TG-PredRNN consistently maintains higher evaluation metrics than the other models, demonstrating good performance. Figure 16 As can be seen, the TG-PredRNN network structure achieved more accurate spatiotemporal sequence prediction results on the MovingMNIST dataset, proving the effectiveness of the transform gate mechanism for general spatiotemporal sequence prediction methods.

[0087] In the SRAD2020 radar echo dataset experiment, the training set consisted of 17,031 images with 811 sequences, and the test set consisted of 3,969 images with 189 sequences. The hyperparameter settings of the network are shown in Table 3.

[0088] Table 3 Hyperparameter settings for SRAD2020 radar echo dataset

[0089]

[0090] The evaluation metrics selected are thresholds τ = 20 and 30, which are used to calculate the evaluation metrics HSS, POD, and CSI. First, the true and predicted values ​​are converted into binary images using the threshold τ. Then, the true positive rate (TP), false negative rate (FN), false positive rate (FP), and true negative rate (TN) of the binary image are calculated based on the pixel values. The expression for HSS is... The expression for CSI is The expression for POD is The experimental results on the SRAD2020 radar echo dataset when τ=20 and τ=30 are shown in Tables 4 and 5.

[0091] Table 4. Results of SRAD2020 radar echo dataset (τ=20)

[0092]

[0093] Table 5. Results of SRAD2020 radar echo dataset (τ=30)

[0094]

[0095] As can be seen from Tables 4 and 5, the average evaluation metrics of TG-PredRNN are significantly higher than those of other network structures. Figure 10 , Figure 11 , Figure 12 , Figure 13 , Figure 14 , Figure 15 The figures show the frame-by-frame evaluation metrics for POD, HSS, and CSI when the thresholds τ = 20 and 30. It can be seen that all evaluation metrics decrease as the number of frames in the predicted sequence increases. The threshold size represents the amount of precipitation; a larger threshold indicates greater precipitation in the region. As the threshold increases, all evaluation metrics decrease. The figures also show that TG-PredRNN consistently outperforms other models in frame-by-frame performance. Figure 17 As can be seen, the TG-PredRNN network structure achieves more accurate prediction results, significantly improving the image quality of radar echo sequence prediction, thus proving the effectiveness of this method in the field of radar echoes.

Claims

1. A radar echo sequence prediction method based on transform gate spatiotemporal long short-term memory units, characterized in that, The specific steps are as follows: Step 1: Use the MovingMNIST dataset and the Standardized RAdar Dataset 2020 radar echo dataset, and divide the datasets into training and testing sets; then preprocess the images in the two datasets to obtain radar echo sequences that can be input into the network structure. Step 2: Based on the spatiotemporal long short-term memory unit, the forget gate is improved by using the transformation gate mechanism, thereby constructing the TG-ST-LSTM structural unit in the network structure. Specifically: In the ST-LSTM cell structure, the forget gate is implemented using the 1-tanh function. The saturation range of the Sigmoid function is compressed, making the forget gate... The output is mapped to the range [0, 0.75] through a transformation gate. After processing by the transformation gate, values ​​close to 1 are reduced to 0.25, values ​​close to 0 are made closer to 1, and values ​​near the middle are compressed to 0.5, thus obtaining the TG-ST-LSTM structural unit, whose expression is as follows: In the formula, Indicates the input modulation gate; Indicates the input gate; Indicates the Gate of Oblivion; These represent the modulation gate, input gate, and forget gate in another set of gate mechanisms, respectively. This indicates the transformation gate mechanism; express Time of the first Layered long-term memory; express Time of the first Layered long-term memory; express Time of the first Layered spatiotemporal memory; express time The spatiotemporal memory of the layers; Indicates the output gate; express Time of the first Layered short-term memory state; tanh represents the tanh function; This represents the Sigmoid function. All represent weights; Both represent bias; Step 3: Obtain the TG-PredRNN network structure by stacking TG-ST-LSTM structural units; specifically: The TG-PredRNN network structure consists of four stacked TG-ST-LSTM structural units; specifically, in the horizontal direction, it transmits the long-term memory state from the previous time step. and hidden state Utilize the hidden state from the previous moment and long-term memory state As the hidden state at the current moment and long-term memory state To enable horizontal interaction and ensure the updating of each hidden state and long-term memory state, including the initial hidden state... and memory state Set to zero; in the vertical direction, pass the hidden state. and spacetime memory units Hidden state and spacetime memory units The hidden states obtained in the last layer are connected together and passed as input to form the TG-PredRNN network structure. and memory unit Feature fusion is performed to obtain the final prediction result; Step 4: Train the TG-PredRNN network structure using the training set data; specifically: The training set images from the MovingMNIST dataset are input into the TG-PredRNN network structure every ten frames. The output results are then optimized using the mean squared error loss function. Image similarity, peak signal-to-noise ratio, and image perceptual similarity are selected as evaluation metrics to evaluate the TG-PredRNN network structure. The training images from the SRAD2020 dataset are input into the TG-PredRNN network structure every ten frames. The output is optimized using the mean squared error loss function, with different thresholds selected. To assess the amount of rainfall, a threshold is used. The values ​​were 20 and 30, respectively, corresponding to different amounts of rainfall; the critical success index, detection probability, and Heidke skill score were selected as evaluation indicators to evaluate the TG-PredRNN network structure. Step 5: Input the test set data into the TG-PredRNN network for testing, then restore the obtained image to finally obtain the predicted result image.

2. The radar echo sequence prediction method based on transform gate spatiotemporal long short-term memory unit according to claim 1, characterized in that, In step 1, specifically: The MovingMNIST dataset was divided into a training set and a test set, with the training set containing 20,000 sequences and the test set containing 5,000 sequences. In the SRAD2020 radar echo dataset, 17,031 consecutive images were used as the training set and 3,969 consecutive images were used as the test set. Then, the size of all images in both datasets was adjusted to 64×64×1, grayscale was performed, and the images were sliced ​​to a 16×16×16 format to reduce the computational cost.

Images