Short-term rainfall prediction method based on multi-modal feature fusion

Abstract

The invention discloses a short-term rainfall prediction method based on multi-modal feature fusion, and the method comprises the following steps: 1, obtaining historical multi-source rainfall data, and carrying out the feature extraction through a feature extraction network, thereby obtaining multi-dimensional original features; 2, performing weighted splicing through a weighted fusion module to obtain initial fusion features; and step 3, the initial fusion feature passes through a weight attention module to obtain a weight coefficient of the multi-dimensional original feature so as to obtain a weight feature of the multi-dimensional original feature, and a multi-modal fusion feature is obtained through splicing. According to the method, when a deep learning technology is applied to a short-term rainfall prediction task, features from different modal data (such as a radar rainfall map and site data) can be processed by using the weighted attention multi-modal fusion mode, so that the accuracy of short-term rainfall prediction is improved.

Description

Technical Field

[0001] The present invention relates to the field of precipitation prediction and deep learning technology in the meteorological field, and specifically refers to a short-term precipitation prediction method based on multimodal feature fusion. Background Art

[0002] Precipitation forecasting plays a key role in meteorological research and is of vital importance to agriculture, water resources management, disaster prevention and mitigation, etc. Accurate prediction of precipitation on a short time scale can significantly improve disaster prevention and mitigation capabilities and reduce the losses caused by floods and waterlogging.

[0003] In recent years, with the rapid development of artificial intelligence technology, short-term precipitation forecasting based on artificial neural networks has become the mainstream method. Artificial neural networks need to analyze and learn relevant historical meteorological data (wind speed, precipitation, cloud cover, etc.) to complete end-to-end rainfall forecasting.

[0004] However, the current mainstream neural network models are facing a difficult problem to solve. Precipitation data is diverse, including but not limited to radar precipitation maps, radar cloud maps, wind speed maps, site data, etc. Data of different modalities are extracted through neural networks to obtain data features of different scales and semantics. According to the current situation, current models often use short links, skip connections, splicing, etc. when using multimodal data. However, the above methods often cause scale mismatch and semantic confusion in the fusion of different modal data, and fail to bring out the advantages of multimodal data. Summary of the invention

[0005] Aiming at the problems of different time scales and semantic confusion in the fusion of different modal data in the prior art, the present invention proposes a short-term precipitation prediction method based on multimodal feature fusion.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A short-term precipitation prediction method based on multimodal feature fusion includes the following steps:

[0008] Step 1: Use the feature extraction network module to extract data features. Taking radar precipitation map and site precipitation data as examples, the former often uses CNN feature extraction network, and the latter often uses GCN feature extraction network. The output feature of the former is M∈R CxHxW , the latter is N∈R CxWxH , C is the image channel; H is the image length; W is the image width.

[0009] Step 2: The features of the two are weightedly spliced ​​through the weighted fusion module designed in this application to obtain the fused initial feature Z.

[0010] Step 3: The initial fusion feature Z is passed through the weighted attention module to obtain the weight coefficient K of the original feature X. 1 , the weight coefficient K of the original feature Y 2 , where K 1 +K 2 =1. The output weight feature is M K =K 1 M and N k =K 2 N.

[0011] Step 4: The original feature branch and the weight branch are concatenated to output the final multimodal fusion feature Z K =M k +N k .

[0012] Preferably, the weighted fusion module calculation process mentioned in step 2 is as follows:

[0013] Z=αM+βN

[0014] Among them, α represents the weight of feature X, and β represents the weight of feature Y. The change of weight is determined by the loss function. The loss function is calculated as follows:

[0015]

[0016] where x i Represents radar precipitation map data, x α Represents the corresponding prediction result. i Indicates the input site data, y α Indicates the corresponding prediction results.

[0017] Preferably, the weighted attention module in step 3 uses a global average pooling layer, an X-axis maximum pooling layer, a Y-axis maximum pooling layer, a Relu activation layer, and a Sigmoid activation layer. The global average pooling layer obtains global information, and the X-axis pooling layer and the Y-axis pooling layer obtain local feature information. After the Relu activation layer, the concatenation is performed and the Sigmoid activation layer is used again to obtain the corresponding weight value K. 1 With K 2 Finally, the weight values ​​are used to calculate the proportion of different modal features. The definition of K is as follows:

[0018]

[0019] Among them, Sig represents the Sigmoid activation layer, Avg represents the global average pooling layer, MaxX represents the maximum pooling layer on the X axis, and MaxY represents the maximum pooling layer on the Y axis. Represents the Relu activation layer.

[0020] As a preference, MSE and RMSE are used as evaluation indicators in the test and evaluation stages, and the calculation method is as follows:

[0021]

[0022] The present invention has the following characteristics and beneficial effects:

[0023] The above technical solution is adopted. This method uses a multimodal feature fusion mechanism based on a weighted attention mechanism. The fusion mechanism proposed in this application can avoid the problem that different scale features cannot be fused and the fusion results of different semantic features are confused during the fusion process of multimodal features. At the same time, in the process of initial feature fusion, a weighted fusion method is used to reduce the information loss and information redundancy problems in the initial splicing, and effectively improve the accuracy of the multimodal feature fusion process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 It is a basic model framework diagram of a multimodal network in an embodiment of the present invention.

[0026] Figure 2 This is an overall flow chart of the multimodal feature fusion mechanism based on the weighted attention mechanism of the present invention.

[0027] Figure 3 This is the flow chart of the weighted attention module in the present invention. DETAILED DESCRIPTION

[0028] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0029] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0030] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0031] The present invention provides a short-term precipitation prediction method based on multimodal feature fusion. When deep learning technology is applied to the short-term precipitation prediction task, the weighted attention multimodal fusion method can be used to process features from different modal data (such as radar precipitation maps, site data), thereby improving the accuracy of short-term precipitation prediction.

[0032] The specific steps are as follows:

[0033] Step 1: Acquire historical multi-source precipitation data, wherein the multi-source precipitation data includes radar precipitation maps and site precipitation data. In this embodiment, the data used are Zhejiang Province radar precipitation maps and Zhejiang Province site data.

[0034] Step 2: Construct a basic multimodal network model framework consisting of a CNN feature extraction network and a GCN feature extraction network. In this embodiment, the radar precipitation map outputs the original feature M∈R through the CNN feature extraction network. CxHxW , the site precipitation data is extracted through the GCN feature network to output the original features N∈R CxWxH , where C is the image channel; H is the image length; and W is the image width.

[0035] It should be noted that the CNN feature extraction network and the GCN feature extraction network are both conventional feature extraction networks, so the method of extracting features by the CNN feature extraction network and the GCN feature extraction network is not specifically described in this embodiment.

[0036] Step 3: Construct a multimodal fusion network mechanism consisting of a weighted fusion module and a weighted attention module.

[0037] Specifically, Figure 2 As shown, in the weighted fusion module, the feature M∈R CxHxW With the feature N∈R CxWxH According to the loss function, weighted fusion is performed to obtain the initial fusion feature Z.

[0038] Z=αM+βN

[0039] Among them, α represents the weight of feature M, β represents the weight of feature N, and the change of weight is determined by the loss function.

[0040] Furthermore, the initial fusion feature Z is passed through the weighted attention module to obtain the weight coefficient K of the original feature M 1 , the weight coefficient K of the original feature N 2 , where K 1 +K 2 =1, the output weight feature is M K =K 1 M and N k =K 2 N, and finally output the final multimodal fusion feature Z through splicing K =M k +N k .

[0041] In this embodiment, the weighted fusion module uses the dual-unit error loss function designed in this application, which is expressed as follows:

[0042]

[0043] where x i Represents radar precipitation map data, x α Represents the corresponding prediction result, y i Indicates the input site data, y α Indicates the corresponding prediction results.

[0044] Further, such as Figure 3 As shown in the figure, the weighted attention module includes a global average pooling layer, an X-axis maximum pooling layer, a Y-axis maximum pooling layer, a Relu activation layer, a Sigmoid activation layer, concat splicing, and multiplication splicing.

[0045] The input features first pass through the global average pooling layer to obtain global information, and the X-axis pooling layer and the Y-axis pooling layer to obtain local feature information. After passing through the Relu activation layer, the concatenation is passed through the Sigmoid activation layer again to obtain the corresponding weight value K 1 With K 2 , and finally use the weight values ​​to calculate the proportion of different modal features.

[0046] The calculation method of the weight value K is as follows:

[0047]

[0048] Among them, Sig represents the Sigmoid activation layer, Avg represents the global average pooling layer, MaxX represents the maximum pooling on the X axis, and MaxY represents the maximum pooling layer on the Y axis. Represents the Relu activation layer.

[0049] Understandably, due to K 1 +K 2 =1, therefore, through the weight value calculation formula, calculate one of the weight values ​​K 1 or K 2 , another weight value can be obtained.

[0050] Step 4: Input the radar precipitation map into the network model architecture composed of steps 2 and 3 to obtain the corresponding results.

[0051] For the network model architecture composed of step 2 and step 3, the experiments in this embodiment are all carried out under the Pytorch deep learning framework. The Pytorch version is 2.0.1, and an NVIDIA 4090 GPU is used for training. The num-workers parameter is set to 32. The weight decay is set to 0.0004. The learning rate is set to 0.1, and the learning rate is adjusted according to the performance changes of the model after every 10 rounds. If the model performance is not significantly optimized, it is adjusted, and if it is normal optimization, it is not adjusted.

[0052] The following are the specific experiments and instructions added:

[0053] This example is compared with several advanced methods on the same dataset, and the results are shown in Table 1.

[0054] Table 1: Comparison with advanced methods on the same dataset

[0055]

[0056] In this embodiment, MSE and RMSE are used as evaluation indicators in the test and evaluation stages, and the calculation method is as follows:

[0057]

[0058] Among them, MSE and RMSE are error indicators, and the lower the MSE, the smaller the model error. It can be seen that the predicted results using the multimodal feature fusion mechanism based on the weighted attention mechanism of the present invention are more accurate and have smaller errors.

[0059] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A short-term precipitation prediction method based on multimodal feature fusion, characterized in that: The steps include: Step 1: Obtain historical multi-source precipitation data, and extract features through a feature extraction network to obtain multi-dimensional original features; Step 2: Perform weighted splicing through the weighted fusion module to obtain the initial fusion features; Step 3: The initial fusion features are passed through the weighted attention module to obtain the weight coefficients of the multi-dimensional original features, and then the weight features of the multi-dimensional original features are obtained, and the multi-modal fusion features are obtained by splicing; Step 4: Analyze the error between the input data and the predicted data, calculate the error index, analyze the model performance based on the input data and the predicted data, and calculate the performance index.

2. The short-term precipitation prediction method based on multimodal feature fusion according to claim 1 is characterized in that: The multi-source precipitation data includes radar precipitation maps and site precipitation data.

3. The short-term precipitation prediction method based on multimodal feature fusion according to claim 1 is characterized in that: The feature extraction network includes a CNN feature extraction network and a GCN feature extraction network, and the radar precipitation map outputs the original feature M∈R through the CNN feature extraction network. CxHxW , the site precipitation data is extracted through the GCN feature network to output the original features N∈R CxWxH , where C is the image channel; H is the image length; and W is the image width.

4. The short-term precipitation prediction method based on multimodal feature fusion according to claim 3 is characterized in that: The weighted fusion method of the multi-dimensional original features is as follows: Z=αM+βN Among them, α represents the weight of feature M, β represents the weight of feature N, and the change of weight is determined by the loss function.

5. The short-term precipitation prediction method based on multimodal feature fusion according to claim 1 is characterized in that: In step 2, the weighted fusion module uses a dual-unit error loss function, which is expressed as follows: where x i Represents radar precipitation map data, x α Represents the corresponding prediction result, y i Indicates the input site data, y α Indicates the corresponding prediction results.

6. The short-term precipitation prediction method based on multimodal feature fusion according to claim 4 is characterized in that: The weighted attention module includes a global average pooling layer, an X-axis maximum pooling layer, a Y-axis maximum pooling layer, a Relu activation layer, a Sigmoid activation layer, concat splicing, and multiplication splicing.

7. The short-term precipitation prediction method based on multimodal feature fusion according to claim 6 is characterized in that: The method for extracting weight features by the weight attention module is: The global average pooling layer obtains global information, and the X-axis pooling layer and the Y-axis pooling layer obtain local feature information. After the Relu activation layer, the concatenation is performed and the Sigmoid activation layer is used again to obtain the corresponding weight value K. 1 With K 2 , and finally use the weight values ​​to calculate the proportion of different modal features.

8. The short-term precipitation prediction method based on multimodal feature fusion according to claim 6 is characterized in that: The calculation method of the weight value K is as follows: Among them, Sig represents the Sigmoid activation layer, Avg represents the global average pooling layer, MaxX represents the maximum pooling on the X axis, and MaxY represents the maximum pooling layer on the Y axis. Represents the Relu activation layer.

9. The short-term precipitation prediction method based on multimodal feature fusion according to claim 8, characterized in that: In step 3, the initial fusion feature Z is passed through the weight attention module to obtain the weight coefficient K of the original feature M 1 , the weight coefficient K of the original feature N 2 , where K 1 +K 2 =1, the output weight feature is M K =K 1 M and N k =K 2 N, and finally output the final multimodal fusion feature Z through splicing K =M k +N k .

Images