Short temporary rainfall prediction method based on multi-modal fusion and dynamic weighting loss

Abstract

The invention discloses a short temporary rainfall prediction method based on multi-modal fusion and dynamic weighting loss. Comprising the following steps: firstly, inputting historical radar and ground station data, and carrying out normalization and downsampling processing; respectively inputting radar and ground station meteorological element data into three layers of networks containing up-sampling and RNN (Recurrent Neural Network) for feature coding; and inputting three layers of networks including fusion, up-sampling and RNN to perform feature fusion and decoding. And finally, outputting accurate rainfall prediction under the constraint of dynamic weighting loss. According to the method, coding, fusion and decoding of radar and ground station meteorological element data are realized. Under the clamping of a fusion network based on an attention mechanism and dynamic weighting loss, the accuracy of short temporary rainfall prediction is improved, and the performance of heavy rainfall prediction is improved more remarkably.

Description

Technical Field

[0001] The invention relates to the technical field of short-term precipitation prediction. Background Art

[0002] Traditional precipitation forecasting mainly relies on numerical weather prediction (NWP). Given the initial conditions and boundary conditions, NWP predicts the future state of the atmosphere by solving the physical equations that describe atmospheric motion through time integration. However, NWP is usually affected by the initial condition field and requires a certain integration time to accelerate the deduction process, which leads to poor short-term prediction results within 0 to 2 hours. In addition, NWP has a high computational cost and is difficult to provide accurate predictions for small-scale areas. Therefore, radar observation-based methods have gradually become an effective alternative for short-term precipitation forecasting. The optical flow method infers the motion of the rainfall area by acquiring the optical flow field in the radar echo map, and has become an important technology for the short-term precipitation forecasting system. However, despite the high computational efficiency of the optical flow method, its accuracy is still low in practical applications, and it is insufficient to deal with complex nonlinear precipitation phenomena.

[0003] Deep learning methods do not rely on complex a priori physical equations and semi-empirical knowledge, but instead infer the evolution of rainfall by training on a large number of samples. This type of method treats short-term precipitation forecasting as a radar video prediction problem. ConvLSTM is currently the leading deep learning model for short-term precipitation forecasting. It combines convolutional neural networks (CNN) with long short-term memory networks (LSTM) to give full play to the advantages of processing spatial and temporal data. Subsequently, researchers proposed a series of deep learning models, including TrajGRU, PredRNN, PredRNN++, MIM, MotionRNN, PredRANN, and RAP-Net. However, although these models have improved performance, their number of parameters, memory usage, and training time have also increased significantly. MS-RNN embeds U-Net into the stacked RNN layer, showing a more powerful spatial and temporal dynamic modeling capability without increasing the number of model parameters.

[0004] However, the above models mainly focus on improving the model structure, ignoring the influence of data characteristics and prediction error characteristics. On the one hand, according to the characteristics of short-term precipitation forecasts, severe convective systems will change significantly within an hour, and it is difficult to make accurate judgments based solely on radar observations. Multimodal learning provides a new solution to this problem, such as MM-RNN, MFSP-Net, FURENet, and MSDM. However, how to effectively integrate multimodal data to achieve accurate precipitation prediction remains a challenge. On the other hand, the prediction difficulty (i.e., error) of each frame changes continuously during the training process, and it is unreasonable to impose the same penalty weight on frames of different prediction difficulties.

[0005] Therefore, how to effectively integrate multi-source data and dynamically allocate loss weights to achieve more accurate short-term precipitation forecasts has become a technical problem that needs to be urgently solved in this field. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a short-term precipitation prediction method based on multimodal fusion and dynamic weighted loss. The method mainly uses radar information and ground station meteorological element data as a supplement to predict precipitation, uses channel and spatial attention mechanism for data fusion, and uses multimodal loss based on dynamic weighting, which can improve the accuracy of various intensities of precipitation prediction, especially heavy precipitation prediction.

[0007] In order to achieve the above-mentioned invention object, the present invention adopts the following technical scheme: a short-term precipitation prediction method based on multimodal fusion and dynamic weighted loss, the method comprising the following steps:

[0008] S1: Input data, input historical radar echo map and gridded ground station meteorological element data.

[0009] S2: Data preprocessing, normalizing and downsampling the input data to obtain processed input data. The specific steps are as follows:

[0010] First, normalize the input data. The formula is as follows:

[0011] Where x represents the current pixel value, min and max represent the minimum and maximum pixel values ​​in the current image respectively. is the normalized pixel value.

[0012] Then, the input grid data is downsampled to 128 x 128.

[0013] S3: Feature encoding, input the radar and gridded ground station meteorological element data into the three-layer network including downsampling and RNN for encoding, and obtain the encoded feature map. The size of the feature map changes during the downsampling process as follows:

[0014] Where C, H, and W represent the number of channels, height, and width of the feature map, respectively.

[0015] S4: Feature fusion and decoding, input the encoded feature map data into a three-layer network including fusion, upsampling and RNN for decoding. The calculation formula of the fusion module is as follows:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024]

[0025] in represents the number of layers, t represents the time step, represents the hidden state of the 0th layer fusion module at the previous moment, R and S represent the characteristic graphs of the radar and ground station meteorological elements, respectively. It is the result of concatenating R, S and F along the channel dimension.

[0026] The size of the feature map changes during upsampling as follows:

[0027]

[0028] S5: Output prediction, output precipitation prediction under the constraint of dynamic weighted loss. First, divide the feature map into multiple blocks in equal proportion, and dynamically generate a loss weight according to the prediction loss of each block, so that blocks with large losses can get higher weights and get more attention during training. At the same time, calculate the prediction loss of fusion mode, radar mode and ground station meteorological element mode to help the fusion module better fuse data. The specific formula is as follows:

[0029]

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036] in represents the mean square error between the predicted value and the true value of the i-th block, p represents the number of blocks cut from a feature map, k, b and x are hyperparameters. m represents the number of input frames, n represents the number of predicted frames, t represents the frame number, (h, w) represents a pixel, , and They represent the weight, true value and predicted value of the (h,w) pixel in the tth frame respectively. Used to ensure that PSNR and BMAE+BMSE are at the same order of magnitude, represents the loss of fusion modality, represents the loss of the radar mode, represents the loss of the meteorological element modality at the ground station, and ML represents the final multimodal loss. Technical Effects

[0037] The present invention provides a short-term precipitation prediction method based on multimodal fusion and dynamic weighted loss, which has at least the following beneficial effects:

[0038] 1) A new multimodal fusion model is proposed for precipitation nowcasting. The model integrates radar echo data and ground station meteorological element data to analyze the spatiotemporal correlation between radar reflectivity and other meteorological elements in rainfall events, thereby achieving more accurate short-term precipitation forecasts.

[0039] 2) A fusion network based on channel and spatial attention mechanisms is proposed. The channel attention mechanism systematically assigns weights to each channel, amplifies important channels, and weakens harmful channels, so that the model can focus on key channels (modalities) more effectively. The spatial attention mechanism assigns weights to spatial positions in the input data, helping the model to prioritize important spatial positions.

[0040] 3) A block-based dynamic weighted loss function is designed to mitigate the impact of prediction error and data imbalance. At different training stages of the model, different weights are dynamically assigned to radar echo blocks with different prediction difficulties, so that the model pays more attention to the difficult-to-predict blocks, thereby ultimately improving the overall prediction performance.

[0041] 4) A new multimodal loss function is designed to maintain the consistency of single modality and fusion modality, enabling the model to effectively fuse multi-source data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a short-term precipitation prediction method based on multimodal fusion and dynamic weighted loss provided by the present invention;

[0043] Figure 2A schematic diagram of the structure of an encoder and a decoder in a multimodal prediction model adopted by the short-term precipitation prediction method provided by the present invention;

[0044] Figure 3 A schematic diagram of the structure of the modal fusion unit F-RNN in the short-term precipitation prediction method provided by the present invention;

[0045] Figure 4 A schematic diagram of block cutting in a dynamic weighted loss function based on block cutting in the short-term precipitation prediction method provided by the present invention; DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings and specific implementation examples.

[0047] Embodiment 1:

[0048] See also Figure 1 — Figure 4 , a short-term precipitation prediction method based on multimodal fusion and dynamic weighted loss, comprising the following steps:

[0049] S1: Input data, input historical radar echo map and gridded ground station meteorological element data;

[0050] S2: Data preprocessing, normalizing and downsampling the input data to obtain processed input data;

[0051] S3: Feature encoding: input the radar and gridded ground station meteorological element data into a three-layer network including downsampling and RNN for encoding to obtain a coded feature map;

[0052] S4: Feature fusion and decoding, the encoded feature map data is input into a three-layer network including fusion, upsampling and RNN for decoding;

[0053] S5: Output prediction, output precipitation prediction under the constraint of dynamic weighted loss.

[0054] In step S2, normalizing and downsampling the input data includes the following steps:

[0055] S21: Normalize the input data. The formula is as follows.

[0056]

[0057] Where x represents the current pixel value, min and max represent the minimum and maximum pixel values ​​in the current image respectively. is the normalized pixel value.

[0058] S22: Downsample the input grid data to 128 x 128.

[0059] In step S3, the radar and gridded ground station meteorological element data are respectively input into a three-layer network including downsampling and RNN for encoding to obtain an encoded feature map; the size of the feature map changes during the downsampling process as follows:

[0060]

[0061] Where C, H, and W represent the number of channels, height, and width of the feature map, respectively.

[0062] In step S4, inputting the encoded feature map data into a three-layer network including fusion, upsampling and RNN for decoding comprises the following steps:

[0063] S41: Modal fusion, fusion module reference Figure 3 , the specific calculation formula is as follows:

[0064]

[0065] in represents the number of layers, t represents the time step, It represents the hidden state of the 0th layer fusion module at the previous moment, and R and S represent the characteristic maps of the meteorological elements of the radar and ground stations, respectively.

[0066] In order to extract important data features, the channel attention module assigns different weights to the data features of each channel. Inspired by SENet, this method implements channel attention in two steps, namely compression and excitation. We introduce an expansion step before compression to promote the interaction between features and capture local and global dependencies between features. Specifically, we implement channel attention in three steps, namely expansion, compression, and excitation. The detailed formula is as follows:

[0067]

[0068]

[0069]

[0070] in It is the result of concatenating R, S and F along the channel dimension.

[0071] Compression is based on the correlation between feature channels. It first generates a weight for each channel, indicating the importance of each feature channel. Then, the features are weighted by element-by-element multiplication to amplify important channels and weaken harmful channels, so that the model can focus on key channels (modalities) more effectively.

[0072] After the channel attention module, the spatial attention module is introduced to assign weights to the spatial positions in the input data, helping the model to prioritize important spatial positions. First, the input feature map is compressed along the channel dimension using average pooling and maximum pooling. The obtained features are concatenated along the channel dimension to form a dual-channel feature map. Subsequently, a convolution operation is applied to ensure that the final feature map is consistent with the input in the spatial dimension. After that, an activation function is applied to obtain the weight coefficient for each position in the feature map. Finally, the weight coefficient is multiplied by the input feature to obtain the scaled new feature. The specific formula is as follows:

[0073]

[0074]

[0075]

[0076]

[0077]

[0078] S42: Upsampling, the size of the feature map changes as follows:

[0079]

[0080] In step S5, the precipitation forecast is output under the constraint of dynamic weighted loss. First, the feature map is divided into multiple blocks in equal proportion, and a loss weight is dynamically generated according to the prediction loss of each block, so that blocks with large losses can get higher weights and receive more attention during the training process. At the same time, the prediction losses of the fusion mode, radar mode and ground station meteorological element mode are calculated to help the fusion module better fuse data.

[0081] It should be noted that in the precipitation prediction task, the prediction difficulty of each frame is different and changes dynamically at different training stages of the model. Assigning the same loss weight to all frames will lead to unreasonable penalties because frames of different difficulties will receive the same penalty, while assigning a fixed weight to each frame will make it difficult for the model to cope with the dynamic changes in prediction difficulty. We need to dynamically assign weights to image frames and pay more attention to frames that are difficult to predict.

[0082] Existing methods mainly focus on dynamically weighting the entire image. However, radar images may contain multiple precipitation areas with different precipitation intensities and prediction difficulties. Directly assigning dynamic weights to the entire image may cause the model to focus too much on areas that are easy to predict. Therefore, we propose a dynamic weighting method based on block cutting, which divides the image into multiple blocks. Each block may correspond to a different precipitation intensity and dynamically assigns a weight according to its prediction difficulty. The block cutting method is as follows: Figure 4shown.

[0083] At different time steps, the difficulty of predicting an image depends on the prediction results of the current model. The greater the difference between the predicted result and the true result, the more difficult the prediction is; the greater the weight loss, the more attention the model pays, and vice versa. The block-based dynamic weighted loss function dynamically assigns loss weights according to the prediction error of each block throughout the training process. This method increases the focus on the difficult-to-predict parts while reducing the focus on the easy-to-predict parts. As a result, the model is better able to focus on more complex precipitation areas, thereby improving its sensitivity in predicting heavy rainfall events. The specific formula is as follows:

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] in represents the mean square error between the predicted value and the true value of the i-th block, p represents the number of blocks cut from a feature map, k, b and x are hyperparameters. m represents the number of input frames, n represents the number of predicted frames, t represents the frame number, (h, w) represents a pixel, , and They represent the weight, true value and predicted value of the (h,w) pixel in the tth frame respectively. Used to ensure that PSNR and BMAE+BMSE are at the same order of magnitude.

[0091] In order to maintain the consistency between unimodal and fused multimodal features, we designed a new multimodal loss (ML) to measure the difference between the prediction losses of unimodal and fused modal features. The key idea is to make the minimum prediction loss based on unimodal features consistent with the prediction loss based on fused modal features. The specific formula is as follows:

[0092]

[0093] in represents the loss of fusion modality, represents the loss of the radar mode, represents the loss of the meteorological element modality of the ground station, a and b are used to set the weight coefficients of the corresponding items, and ML represents the final multimodal loss.

[0094] The following takes a specific embodiment as an example to further illustrate this specific implementation method. Specific embodiment 1

[0096] The validation dataset for this method is the multimodal dataset RAIN-F provided by the Korea Meteorological Administration (KMA). This dataset includes radar, ground-site meteorological elements, and satellite data, and is a multimodal dataset that is aligned in time and space. The temporal coverage ranges from 2017 to 2019, and the spatial coverage is the Korean Peninsula (126.3°-129.3°E, 35°-38°N). Specifically, it includes nine atmospheric state variables related to precipitation (1 radar, 7 ground-site meteorological elements, and 1 satellite). The temporal resolution is 1 hour, and the spatial resolution is different for each variable. Radar products are provided by the Korea Meteorological Administration, which manages a network of S-band weather radars. The Open Radar Product Generator (ORPG) and fuzzy quality control algorithms are used to control the quality of radar data. Since satellite data has a lower resolution and is less accurate than radar, this data modality is not used. The previous version of the dataset included 7 ground-site meteorological element data: wind direction, wind speed, humidity, surface pressure, sea level pressure, temperature, and rainfall. However, the revised new version of the dataset excludes two meteorological elements with serious data missing, namely, surface pressure and sea level pressure. The present invention uses a revised version of RAIN-F, which uses radar data and five ground variables. The statistics of the precipitation rate distribution of the dataset are shown in Table 1.

[0097] Table 1 Statistics of precipitation rates in the RAIN-F dataset

[0098] Precipitation rate (mm / h) Proportion (%) Precipitation level [0, 0.5) 94.33 Light rain [0.5, 2) 3.99 light rain [2, 5) 1.15 light to moderate rain [5, 10) 0.36 moderate rain [10, +∞) 0.17 Moderate to heavy rain

[0099] The precipitation prediction evaluation index is the CSI score, and the formula is as follows.

[0100]

[0101] Among them, TP represents the number of correctly predicted grids, FN represents the number of missed grids, and FP refers to the number of false positive grids.

[0102] All input images are uniformly interpolated to 128×128 pixels. All RNNs use 3×3 convolution kernels. We use the PyTorch framework, Adam optimizer, and set the batch size to 8. On the Rain-F dataset, we use 4 consecutive frames as a batch, of which 2 frames are used as input and 2 frames are used as output. The number of channels in the hidden state is 48, and the initial learning rate is set to 0.00003. Hyperparameters in multimodal loss , , , , and .

[0103] It should be noted that increasing the number of blocks does not necessarily lead to better performance, and more blocks will lead to increased time and space overhead because more dynamic weights need to be calculated. In the experiment, we found that the optimal number of blocks on the RAIN-F dataset is 2.

[0104] Encoder and decoder structure reference Figure 2 ; The structure of the fusion unit F-RNN refers to Figure 3 The specific parameters are shown in Table 2.

[0105] Table 2 Specific parameters of the fusion unit F-RNN

[0106]

[0107] The selected baseline models include the single-modal model MS-RNN and the multi-modal model MM-RNN. These models can adapt to various architectures, such as ConvLSTM, PredRNN, PredRNN++ and MIM, and are lighter than traditional stacked structures. Similarly, the multi-modal fusion model MMF-RNN proposed in the present invention can also adapt to these architectures. Table 3 below lists the prediction performance of the three on ConvLSTM and PredRNN++.

[0108] Table 3 Comparison of prediction performance of the model implemented in the present invention with other methods

[0109]

[0110] It can be seen that our model MMF-RNN performs best in various performance indicators. Specifically, compared with the second best model, the BMSE scores of MMF-ConvLSTM and MMF-PredRNN++ decreased by 13.73% and 10.32%, and the BMA scores decreased by 7.85% and 4.79%, respectively. In addition, it significantly outperformed other models in predicting heavy precipitation (R ≥ 5 and R ≥ 10). Compared with the strongest MM-RNN model, MM-PredRNN++, the CSI (R ≥ 5) and CSI (R ≥ 10) of MMF-PredRNN++ increased from 0.167 and 0.022 to 0.264 and 0.13, respectively, an increase of 58.08% and 490.90%.

[0111] It should be noted that the above embodiments are only preferred examples of the present invention and are not intended to limit the protection scope of the present invention. Any equivalent replacement or substitution based on the above technical solution should be considered within the protection scope of the present invention.

Claims

1. A short-term precipitation prediction method based on multimodal fusion and dynamic weighted loss, characterized in that: The following steps are involved: S1: Input data, input historical radar echo map and gridded ground station meteorological element data; S2: Data preprocessing, normalizing and downsampling the input data to obtain processed input data; S3: Feature encoding: input the radar and gridded ground station meteorological element data into a three-layer network including downsampling and RNN for encoding to obtain a coded feature map; S4: Feature fusion and decoding, the encoded feature map data is input into a three-layer network including fusion, upsampling and RNN for decoding; S5: Output prediction, output precipitation prediction under the constraint of dynamic weighted loss.

2. The method for predicting short-term precipitation based on multimodal fusion and dynamic weighted loss according to claim 1, characterized in that: In step S2, normalizing and downsampling the input data includes the following steps: S21: Normalize the input data. The formula is as follows: Where x represents the current pixel value, min and max represent the minimum and maximum pixel values ​​in the current image respectively. is the normalized pixel value. S22: Downsample the input grid data to 128 x 128.

3. The method for predicting short-term precipitation based on multimodal fusion and dynamic weighted loss according to claim 1, characterized in that: In step S3, the radar and gridded ground station meteorological element data are respectively input into a three-layer network including downsampling and RNN for encoding to obtain an encoded feature map; the size of the feature map changes during the downsampling process as follows: Where C, H, and W represent the number of channels, height, and width of the feature map, respectively.

4. The method for predicting short-term precipitation based on multimodal fusion and dynamic weighted loss according to claim 1, characterized in that: In step S4, inputting the encoded feature map data into a three-layer network including fusion, upsampling and RNN for decoding comprises the following steps: S41: Modal fusion, the calculation formula of the fusion module is as follows: in represents the number of layers, t represents the time step, represents the hidden state of the 0th layer fusion module at the previous moment, R and S represent the characteristic graphs of the radar and ground station meteorological elements, respectively. It is the result of concatenating R, S and F along the channel dimension. S42: Upsampling, the size of the feature map changes as follows:

5. The method for predicting short-term precipitation based on multimodal fusion and dynamic weighted loss according to claim 1, characterized in that: In step S5, the precipitation forecast is output under the constraint of dynamic weighted loss. First, the feature map is divided into multiple blocks in equal proportion, and a loss weight is dynamically generated according to the prediction loss of each block, so that blocks with large losses can get higher weights and receive more attention during the training process. At the same time, the prediction losses of the fusion mode, radar mode, and ground station meteorological element mode are calculated to help the fusion module better fuse data. The specific formula is as follows: in represents the mean square error between the predicted value and the true value of the i-th block, p represents the number of blocks cut from a feature map, k, b and x are hyperparameters. m represents the number of input frames, n represents the number of predicted frames, t represents the frame number, (h, w) represents a pixel, , and They represent the weight, true value and predicted value of the (h,w) pixel in the tth frame respectively. Used to ensure that PSNR and BMAE+BMSE are at the same order of magnitude, represents the loss of fusion modality, represents the loss of radar mode, represents the loss of the meteorological element modality at the ground station, and ML represents the final multimodal loss.

Images