Short-term precipitation prediction method based on spatiotemporal attention and data fusion

Abstract

The invention relates to a short-term precipitation prediction method based on spatiotemporal attention and data fusion. Firstly, multi-source heterogeneous data are input and preprocessed. Secondly, the denoised data are input into a spatiotemporal diffusion module based on attention, and different spatiotemporal resolution data of different modes are diffusely aligned in time and space scales. Then, the fusion of high-dimensional semantics and low-dimensional semantics is realized through an adaptive attention fusion module. Then, the aligned historical radar echo map sequence and the pattern prediction data sequence are respectively input into a radar echo map encoder and a pattern data encoder including a convolution layer and a ConvLSTM module for data encoding. Then, the encoded radar echo map data and the pattern data are convoluted to change the number of characteristic channels of the data to obtain fused data. Finally, the data stream obtained by the radar echo encoder and the fused data stream are input into a decoder including a convolution layer and a ConvLSTM module for data decoding to obtain a predicted radar echo map sequence and perform Z-R transformation to obtain a precipitation prediction result.

Description

Technical Field

[0001] The present invention belongs to the technical field of weather forecasting, and in particular relates to a short-term precipitation prediction method based on multi-data fusion and deep learning prediction. Background Art

[0002] Traditional short-term precipitation forecasting methods can be summarized into three categories: one is based on statistical methods, such as physical statistical model methods; the second is based on model numerical prediction; the third is based on radar charts. However, due to the randomness of the weather system, precipitation is affected by multiple factors such as climate zones, geographical altitudes, and monsoon circulation, making the trend of short-term precipitation highly nonlinear. Traditional methods cannot handle the nonlinear relationship in short-term precipitation well.

[0003] Artificial neural networks have been applied to the field of weather forecasting in recent years. They can model nonlinear systems, use deep learning, extract low-level features through multi-layer network structures and nonlinear transformations, and thus discover the connection between climate characteristics and climate phenomena. Traditional neural network methods are mainly based on simple convolutional neural networks and recurrent neural networks. They often consider the changes in meteorological data and make predictions at a single time point or a single spatial location. They do not consider that meteorological data has a global spatiotemporal relationship. Therefore, there is an urgent need for a technology that can model time and space at the same time. The attention mechanism, which has received widespread attention in recent years, can be used to measure the correlation between input data. Existing methods focus on the application of the attention mechanism in image processing, but do not consider the possibility of applying it to spatiotemporal prediction problems. In addition, traditional neural network methods mostly use single-mode input, that is, the input data is one of the data such as radar echo map, cloud distribution, humidity distribution, etc., so they cannot make good use of huge data to improve the prediction ability of short-term precipitation. Therefore, a technology that can improve the accuracy of short-term precipitation forecast through multi-modal input is needed. In addition, multimodal data often have inconsistent temporal and spatial resolutions. The traditional method of modifying spatiotemporal resolution with variable weights based on artificial experience cannot effectively perform differential spatiotemporal expansion according to the differences in time and space, which is not conducive to subsequent feature extraction and data fusion. Summary of the invention

[0004] The present invention aims at the technical problems existing in the prior art and provides a short-term precipitation prediction method based on spatiotemporal attention and data fusion. This method not only helps to improve the accuracy of short-term precipitation forecast, which has practical significance for industries such as agriculture, transportation, and electricity, but also improves the efficiency and depth of data mining for multi-source heterogeneous information, and solves the problem of multi-modal input and data fusion.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows: a short-term precipitation prediction method based on spatiotemporal attention and data fusion, which comprises the following steps:

[0006] S1: input data and preprocess the data;

[0007] S2: The denoised data is input into the attention-based spatiotemporal diffusion module to align the data of different modalities in time and space, and then the fusion of high-dimensional semantics and low-dimensional semantics is realized through the adaptive attention fusion module;

[0008] S3: inputting the aligned radar echo image sequence and pattern data sequence into a radar echo image encoder and a pattern data encoder including a convolutional layer and a ConvLSTM module respectively for data encoding, thereby obtaining encoded radar echo image data and pattern data;

[0009] S4: The encoded radar echo data and pattern data are convolved to change the number of feature channels of the data, and feature fusion is performed by residual addition to obtain fused data;

[0010] S5: The data stream obtained by the radar echo encoder and the fused data stream are input into the decoder including the convolution layer and the ConvLSTM module for data decoding, and the predicted radar echo map sequence is obtained and ZR transformation is performed to obtain the short-term precipitation prediction result sequence and output it, thus completing the short-term precipitation prediction task.

[0011] In the above step S1, the preprocessing of the input radar echo map sequence and the pattern data sequence includes the following steps: the original radar echo map and the pattern data are subjected to wavelet denoising with a hard threshold function to perform data denoising. The hard threshold wavelet denoising can well preserve local features such as signal edges, and can better preserve key parts and remove noise points compared to other denoising methods.

[0012] The process of encoding the radar echo pattern encoder and the pattern data in the above step S2 is as follows:

[0013] Since the resolution of the input radar echo map is 1km*1km and the time interval is 12min, while the resolution of the mode data is 3km*3km and the time interval is 60min, it causes a mismatch in time and space. The present invention proposes to use a convolutional neural network with a spatiotemporal attention mechanism to align data of different modes in time and space. First, the denoised data is input into the attention-based spatiotemporal diffusion module to align the data of different modes in time and space scales; then the aligned data is input into the convolutional network encoder to extract features while adjusting the resolution and the number of channels; finally, the extracted high-dimensional and low-bit information is fused through the adaptive attention fusion module; since the weights are shared when processing different inputs, it can adapt to inputs of different sizes and perform effective fusion. By designing the attention-based spatiotemporal diffusion module and the adaptive attention fusion module, effective features are extracted from the input data, which is more in line with the distribution characteristics of the data than the ordinary interpolation method. The scheme can realize dynamic interpolation, does not rely on a fixed interpolation method, and has greater flexibility and adaptability for different input data. The spatiotemporal attention mechanism is used to mine the dependency of the input data on the spatiotemporal scale, thereby extracting more valuable features.

[0014] The process of encoding the radar echo image encoder and the pattern data in the above step S3 is as follows: the convolution layers of the radar echo image encoder and the pattern data encoder both use a 3×3 convolution kernel to perform convolution operations on the image, and the ConvLSTM modules of the encoder both use a 5×5 convolution kernel.

[0015] In the data fusion process in step S4 above, the 3×3 convolution kernels of the hidden state output by the ConvLSTM of the same encoder and the same level are superimposed, and the residual structure directly superimposes the hidden state output by the radar echo map encoder on the superposition result. This solution can realize data fusion at different levels through convolution kernels of different sizes, thereby improving the strength and breadth of data fusion.

[0016] In the decoding process in step S5 above, the hidden state of the three-layer ConvLSTM module in the decoder adopts the hidden state obtained by data fusion processing, and the hidden state obtained by data fusion is input into the decoder in the order of the pre-order scheme; the generation model uses the linear interpolation method to enlarge the size of the feature map. The two convolution layers in the decoder use 3*3 convolution kernels and 1*1 convolution kernels respectively. The pixel value of the radar echo image is based on the relationship Calculate the logarithmic radar reflectivity factor Then, according to the relationship dBz=10loga+10b logR, the precipitation intensity R is calculated, where the constants a=58.53, b=1.56, to complete the short-term precipitation forecast task.

[0017] Compared with the prior art, the advantages of the present invention are as follows:

[0018] 1) The precipitation prediction method proposed in the present invention fuses radar echo images and numerical forecast model data, aligns data of different modes in terms of time and space through a convolutional neural network with a spatiotemporal attention mechanism, and constructs a sequence-to-sequence model through ConvLSTM, which greatly improves the prediction accuracy of short-term precipitation weather;

[0019] 2) The attention-based spatiotemporal diffusion module and adaptive attention fusion module proposed in the present invention can realize dynamic interpolation, do not rely on fixed interpolation methods, have greater flexibility and adaptability for different input data, and can mine the dependency of input data on the spatiotemporal scale through the spatiotemporal attention mechanism, thereby extracting more valuable features;

[0020] 3) The present invention proposes to apply hard threshold wavelet denoising in the data processing stage. Wavelet denoising can well preserve local features such as signal edges. Compared with other denoising methods, it can better preserve key parts and remove noise points, thereby improving the accuracy of short-term precipitation forecast;

[0021] 4) The convolutional fuser with a residual structure proposed in the present invention can realize data fusion at different levels through convolutional nuclei of different sizes, improve the strength and breadth of data fusion, and enhance the accuracy of short-term precipitation forecast. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is the spatiotemporal attention mechanism in step S1;

[0023] Figure 2 is a schematic diagram of step S3 and step S4;

[0024] Figure 3 Flowchart of the short-term precipitation prediction method based on spatiotemporal attention and data fusion. Specific implementation methods

[0025] In order to deepen the recognition and understanding of the present invention, the present application is further described below in conjunction with the accompanying drawings and specific embodiments.

[0026] The short-term precipitation prediction method based on spatiotemporal attention and data fusion includes the following steps:

[0027] S1: First input the radar echo sequence at time t (V1, V2, ..., V t ) and the pattern data sequence (K1, K2, ..., K t ), perform wavelet threshold denoising on the input data;

[0028] S2: Secondly, the denoised data is input into the attention-based spatiotemporal diffusion module to align the data of different modalities in time and space. Then, the fusion of high-dimensional semantics and low-dimensional semantics is achieved through the adaptive attention fusion module;

[0029] S3: Then, the aligned radar echo image sequence and pattern data sequence are respectively input into a radar echo image encoder and a pattern data encoder including a convolutional layer and a ConvLSTM module for data encoding to obtain encoded radar echo image data and pattern data.

[0030] S4: Next, the encoded radar echo map data and pattern data are convolved to change the number of feature channels of the data, and feature fusion is performed by residual addition to obtain fused data;

[0031] S5: Finally, the data stream obtained by the radar echo encoder and the fused data stream are input into the decoder including the convolution layer and the ConvLSTM module for data decoding to obtain the predicted radar echo map sequence and perform ZR transformation to obtain the precipitation prediction result.

[0032] Among them, the two spatiotemporal attention structures proposed in this scheme are as follows Figure 1 As shown, they are temporal attention and spatial attention respectively. In temporal attention, the input data is average pooled and max pooled respectively, and then the results are respectively extracted features through the convolution layer, and finally the residuals are added as output, so that more attention can be paid to the temporal correlation of the input. In spatial attention, the input is directly spliced ​​together after average pooling and max pooling, and then the features are extracted through the convolution layer, and finally the residuals are added, so that more attention is paid to spatial correlation. The two fusion modules used in this patent are the attention-based spatiotemporal diffusion module and the adaptive attention fusion module, and their detailed structures are as follows:

[0033] The attention-based spatiotemporal diffusion module includes a spatial diffusion module and a temporal diffusion module. First, the temporal attention, spatial attention and convolutional layers are combined into a spatiotemporal attention module by means of hierarchical residual connection and stacking. Secondly, since the input pattern data is relatively sparse compared to the radar echo data, the spatial resolution is first amplified in the spatial diffusion module, so two convolutional layers are connected after the spatiotemporal attention module to increase the spatial resolution from 3KM to 1KM; and the spatially amplified data still needs temporal augmentation, so convolution processing is first performed in the temporal diffusion module, and then two spatiotemporal attention modules are connected to increase its temporal resolution from 1 hour to 12 minutes.

[0034] The adaptive attention fusion module consists of a 1*1 convolution and a spatiotemporal attention module. The input pattern data is divided into high-level semantic information and low-level semantic information due to the different number of convolution layers. The adaptive attention fusion module can fuse the extracted high-dimensional and low-level information. First, the input data is input into the spatiotemporal attention module, and then it is input into the 1*1 convolution to adjust the number of feature channels. Since the weights are shared when processing different inputs, it can adapt to inputs of different sizes and perform effective fusion.

[0035] This solution uses the ConvLSTM module to process spatiotemporal information. The ConvLSTM module is a convolution-based neural network structure. The difference between it and LSTM is that the matrix multiplication in LSTM is changed to a convolution layer, which can be used to process spatiotemporal sequence information.

[0036] The decoder part generally includes the following two modules: ConvLSTM module and convolution layer. The hidden state of the three-layer ConvLSTM module in the decoder adopts the hidden state obtained by data fusion processing, and the hidden state obtained by data fusion is input into the decoder in the order of the pre-order scheme; the two convolution layers in the decoder use 3*3 convolution kernels and 1*1 convolution kernels respectively. The convolution layer in the decoder is used to restore the small-size feature map to the full-size echo image. It is an important component for generating prediction results. Linear interpolation plus convolution is used instead of deconvolution. This method consists of two parts: 1) Use general upsampling methods (nearest neighbor, bilinear interpolation, etc.) to change the image size and enlarge the feature map. 2) Convolution is performed on the basis of the enlarged feature map obtained in step 1). The pixel value of the radar echo image is based on the relationship Calculate the logarithmic radar reflectivity factor Then, according to the relationship dBz=10loga+10b logR, the precipitation intensity R is calculated, where the constants a=58.53, b=1.56, to complete the short-term precipitation forecast task.

[0037] Example 1: The data set for verification of this method is the radar echo map and pattern data provided by the Guangdong Provincial Meteorological Bureau. Among them, the geographical range of the radar echo map is South China, and the data unit dBZ represents the radar echo intensity, which is generally within the range of 0-80dBZ. The longitude span is 107°E-119°E. The latitude span is 18°N-27°N. The time span is from March 2017 to December 2018. The resolution is 1 km. The data interval is 12 minutes. The ZR relationship represents the relationship between reflectivity Z and precipitation intensity R (mm / h), where dBZ=10log 10 a+10blog 10R, a, and b are the parameters of the radar itself. In this experiment, the values ​​are: a = 58.53, b = 1.56. dBZ is commonly used to describe precipitation. Generally, the larger the value, the greater the precipitation. The spatial range intercepted in this experiment is: 108.6°E-117.6°E, 18.0°N-27.0°N, with a resolution of 3 kilometers. Here 0.01°≈1 kilometer. In addition, the 23 eigenvalues ​​of the model data, including humidity at different heights (9) and wind speed (12), are provided by GRAPES in South China. The matrix size is 100*100*23, with a resolution of 3 kilometers and 1 hour. Combined with experimental experience, this experiment uses the radar echo map of the first 10 moments and the gridded temperature total precipitation to predict the radar echo map of the last 10 moments.

[0038] The precipitation prediction evaluation index in the meteorological field is the CSI score:

[0039]

[0040] Where TP is the number of correctly predicted grid points, FN is the number of missed grid points, and FP is the number of falsely reported grid points. This experiment aims to improve the CSI score of the grid sequence for precipitation prediction. The r value is the rainfall intensity level obtained after Z-RC transformation, and its corresponding relationship with precipitation broadcast is shown in Table 1.

[0041] The corresponding relationship between the r value and the precipitation report in Table 1 is as follows

[0042] R value gear Precipitation forecast r≥0.5 Light rain r≥2 light rain r≥5 moderate rain r≥10 heavy rain r≥30 rainstorm

[0043] Traditional methods usually use radar echo maps or pattern data to make predictions separately, but we use the data fusion model to build a deep learning neural network model. Table 2 shows the average scores of several models for the CSI index of the experimental data at different levels. The higher the CSI index, the higher the precipitation prediction accuracy. As can be seen from the table below, this method has a significant improvement in prediction scores compared with traditional methods.

[0044] Table 2 Comparison of prediction scores of the model implemented in the present invention with other methods

[0045]

[0046] Embodiment 2: This embodiment discloses another data fusion method based on the spatiotemporal attention mechanism. In this embodiment, data fusion no longer occurs in the complete stage of the three-layer encoder, but only the last layer of encoder is fused, which can be regarded as a simplified version of Embodiment 1. The purpose of this is to reduce the computational complexity while retaining as much prediction accuracy as possible. The data set used in this embodiment is the radar echo map and pattern data provided by the Guangdong Provincial Meteorological Bureau. The longitude span is 107°E-119°E. The latitude span is 18°N-27°N. The time span is from March 2017 to December 2018. The resolution is 1 km. The data interval is 12 minutes. When the number of fusion parameters is reduced to one-third of the original, the precipitation forecast CSI score for different r values ​​is reduced by an average of 11%.

[0047] It should be noted that the above embodiments are not intended to limit the protection scope of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. A short-term precipitation prediction method based on spatiotemporal attention and data fusion, the method comprising the following steps: S1: input data and preprocess the data; S2: The denoised data is input into the attention-based spatiotemporal diffusion module to align heterogeneous data of different modalities in time and space, and then the fusion of high-dimensional semantics and low-dimensional semantics is achieved through the adaptive attention fusion module; Wherein, step S2 mainly includes the following steps: S21: Two spatiotemporal attention structures are proposed, namely temporal attention and spatial attention. In temporal attention, the input data is average pooled and max pooled respectively, and then the results are respectively passed through the convolution layer to extract features, and finally the residuals are added as output. In spatial attention, the input is average pooled and max pooled, and then directly concatenated and features are extracted through the convolution layer, and finally the residuals are added; S22: The attention-based spatiotemporal diffusion module includes a spatial diffusion module and a temporal diffusion module. First, the temporal attention, spatial attention and convolutional layers are combined into a spatiotemporal attention module by means of hierarchical residual connection and stacking. Secondly, the spatial resolution of the spatial diffusion module is first amplified, and then two convolutional layers are connected after the spatiotemporal attention module to increase the spatial resolution from 3KM to 1KM; and the spatially amplified data is first convolved in the temporal diffusion module, and then connected to two spatiotemporal attention modules to increase its temporal resolution from 1 hour to 12 minutes. S23: The adaptive attention fusion module consists of a 1*1 convolution and a spatiotemporal attention module. The input data is first fed into the spatiotemporal attention module, and then fed into a 1*1 convolution to adjust the number of feature channels and fuse high-level semantic information with low-level semantic information. S3: inputting the aligned historical radar echo map sequence and the pattern prediction data sequence into a radar echo map encoder and a pattern data encoder including a convolutional layer and a ConvLSTM module respectively for data encoding, thereby obtaining encoded radar echo map data and pattern data; S4: The encoded historical radar echo map data and the mode prediction data are convolved to change the number of feature channels of the data, and feature fusion is performed by residual addition to obtain fused data; S5: The data stream obtained by the radar echo encoder and the fused data stream are input into the decoder including the convolution layer and the ConvLSTM module for data decoding to obtain the predicted radar echo map sequence and perform ZR transformation to obtain the precipitation prediction result.

2. The short-term precipitation prediction method based on spatiotemporal attention and data fusion according to claim 1 is characterized in that: In step S1, the radar echo map sequence (V1, V2, ..., V t ) and the corresponding time pattern data sequence (K1, K2, ..., K t ), perform wavelet threshold denoising and interpolation on the input data.

3. The short-term precipitation prediction method based on spatiotemporal attention and data fusion according to claim 1, characterized in that: In step S3, the convolution layers of the radar echo image encoder and the pattern data encoder both use a 3×3 convolution kernel to perform a convolution operation on the image, and the ConvLSTM modules of the encoder both use a 5×5 convolution kernel.

4. The short-term precipitation prediction method based on spatiotemporal attention and data fusion according to claim 1, characterized in that: In the step S4, the hidden states of the ConvLSTM output of the radar echo image encoder and the pattern data at the same level are superimposed through a 3×3 convolution kernel, and the hidden states of the radar echo image encoder output are directly superimposed on the result through a residual connection.

5. The short-term precipitation prediction method based on spatiotemporal attention and data fusion according to claim 1, characterized in that: In step S5, before inputting the data into the convolution layer, the size of the feature map is first enlarged by the linear interpolation method. The two convolution layers use 3*3 convolution kernels and 1*1 convolution kernels to process the enlarged feature map respectively. The hidden state of the three-layer ConvLSTM module in the decoder adopts the hidden state obtained by data fusion processing, and the hidden state obtained by data fusion is input into the decoder according to the encoding order. The pixel value of the radar echo image is calculated according to the relationship Calculate the logarithmic radar reflectivity factor Then, according to the relationship dBz=10loga+10blogR, the precipitation intensity R is calculated, where the constants a=58.53, b=1.56, and the short-term precipitation forecast task is completed.

Images