A short-term precipitation prediction method, device and equipment based on a multi-modal RNN

Abstract

The application discloses a short-term and nowcasting precipitation prediction method based on a multimodal RNN, relates to the technical field of short-term and nowcasting precipitation prediction, and aims to provide a short-term and nowcasting precipitation prediction method with high accuracy, which comprises the following steps: acquiring a radar graph and a meteorological element graph; training a multimodal model by using the radar graph and the meteorological element graph; and performing precipitation prediction by using the trained multimodal prediction model; wherein the multimodal prediction model comprises a radar module and a meteorological element module, and the multimodal prediction model obtains a short-term and nowcasting precipitation prediction result according to a radar prediction graph; the method takes radar information as the main part and meteorological element information as the auxiliary part to perform precipitation prediction, fuses the two modes, and improves the accuracy of short-term and nowcasting precipitation prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of short-term precipitation prediction. BACKGROUND

[0002] Traditional precipitation prediction is mainly based on numerical weather prediction (NWP). Given initial conditions and boundary conditions, NWP solves physical equations describing atmospheric motion by time integration to predict future atmospheric state. According to the local climate characteristics, many advanced meteorological service centers in the world have gradually established their own numerical prediction models, such as GRAPES in China and WRF in the United States. However, NWP is usually affected by the initial condition field and needs a certain integration period to accelerate the evolution process, which leads to poor prediction results within 0 to 2 hours of advance time. In addition, NWP has high computational cost and cannot provide small-scale prediction. Therefore, radar observation-based prediction has become an alternative method for short-term precipitation prediction. The optical flow method infers the motion of the rainfall area by obtaining the optical flow field of the radar echo map, and has become the mainstream of very short-time precipitation prediction system. However, although the optical flow method has high computational efficiency, its accuracy is low in practical application and cannot cope with complex non-linear changes in precipitation.

[0003] Deep learning methods do not rely on complex prior physical equations and semi-empirical knowledge, but infer the evolution of rainfall by training a large number of samples. Such methods treat short-term precipitation prediction as a radar video prediction problem. Currently, video prediction models are mainly based on Markov assumption recurrent neural networks (RNN). Since ConvLSTM introduced convolution into LSTM to capture the spatio-temporal changes of precipitation, a series of subsequent models are based on it, constantly refreshing the prediction performance, and their evolution can be roughly described as: ConvLSTM, TrajGRU, PredRNN, PredRNN++, MIM, MotionRNN, and finally MS-RNN. However, they all focus on improving the model structure, without considering the influencing factors at the data level. That is, these models mainly treat short-term precipitation prediction as a general video prediction task (such as human or car trajectory prediction), without fully utilizing meteorological characteristics. In fact, precipitation is a complex microphysical process affected by humidity, temperature, terrain, and other factors in the environment. It is unreasonable to consider only a single radar modality without considering the changes of other elements in the atmospheric system, and the prediction results obtained in this way are inaccurate.

[0004] Therefore, how to provide a short-term precipitation prediction method with high accuracy has become a technical problem to be solved in the field. SUMMARY

[0005] In order to solve the technical problem of improving the short-term precipitation prediction accuracy, the present application provides a short-term precipitation prediction method based on a multi-modal RNN, a device and equipment, which takes radar information as the main and meteorological element information as the auxiliary to carry out precipitation prediction, and fuses the two modes to improve the accuracy of short-term precipitation prediction.

[0006] Based on the same inventive concept, the present application has three independent technical solutions:

[0007] 1. A short-term precipitation prediction method based on a multi-modal RNN, the method comprising:

[0008] obtaining a radar map and a meteorological element map;

[0009] training a multi-modal RNN model using the radar map and the meteorological element map;

[0010] carrying out precipitation prediction using the trained multi-modal RNN model;

[0011] wherein the multi-modal RNN model comprises a radar module and a meteorological element module, and the multi-modal RNN model obtains a short-term precipitation prediction result according to a predicted radar map.

[0012] Further, the prediction process of the multi-modal RNN model is:

[0013] inputting the radar map and the meteorological element map at the same time into the radar module and the meteorological element module respectively to form a radar flow and a meteorological element flow, and the input of the radar module further comprises a meteorological element flow from the meteorological element module;

[0014] the radar module outputs a predicted radar map, and the meteorological element module outputs a predicted meteorological element map.

[0015] Further, the meteorological element module and the radar module are both multi-scale RNN models, comprising three encoders and three decoders, the encoders adopt a pooling operation, and the decoders adopt an up-sampling operation.

[0016] Further, the meteorological element flow sequentially passes through a first encoder, a second encoder, a third encoder, a first decoder, a second decoder and a third decoder, the meteorological element flow of the first encoder further flows to the third decoder, and the meteorological element flow of the second encoder further flows to the second decoder; and the third decoder outputs a predicted meteorological element map.

[0017] Further, the radar flow sequentially passes through a fourth encoder, a fifth encoder, a sixth encoder, a fourth decoder, a fifth decoder and a sixth decoder, the radar flow of the fourth encoder further flows to the sixth decoder, and the radar flow of the fifth encoder further flows to the fifth decoder;

[0018] The meteorological element stream of the first decoder also flows to the fourth decoder, the meteorological element stream of the second decoder also flows to the fifth decoder, and the meteorological element stream of the third decoder also flows to the sixth decoder; the sixth decoder outputs a predicted radar map.

[0019] Further, the radar module further comprises a modal fusion unit, configured to fuse the meteorological element features and the radar features by using an attention mechanism.

[0020] The modal fusion unit is calculated according to the following formula:

[0021]

[0022]

[0023] Q = (W q R) T ,K = W k E,V = W V E,

[0024] S = softmax(QK),

[0025] A = VS T ,

[0026] F = A + R + E

[0027] wherein R is a radar feature, E is a meteorological element feature, K is a key of the modal fusion unit, V is a value of the modal fusion unit, Q is a query variable of the modal fusion unit, S is a similarity score matrix, A is an attention matrix, W k , W q , W V are parameters of K, Q and V respectively, F is a result obtained after modal fusion, ED i (i = 0, 1, 2) represent the decoders in the meteorological element module, RE i and RD i (i = 0, 1, 2) represent the encoders and decoders in the radar module respectively.

[0028] Further, the loss function of the multi-modal RNN model is represented as follows:

[0029]

[0030]

[0031] L total = λ1L ele + λ2L rad ,

[0032] wherein Ltotal L is a model loss function ele L is a meteorological element loss function rad L is a radar loss function, m+n is the total length of the radar map sequence or the meteorological element map sequence, λ1 and λ2 are respectively a meteorological element loss weight and a radar loss weight, R is a predicted radar map t R is a real radar map E is a predicted meteorological element map t E is a real meteorological element map.

[0033] 2. A short-term precipitation prediction device based on a multi-modal RNN, comprising:

[0034] a picture acquisition module configured to acquire a radar map and a meteorological element map;

[0035] a training module configured to train a multi-modal RNN model using the radar map and the meteorological element map;

[0036] a prediction module configured to perform precipitation prediction using the trained multi-modal RNN model;

[0037] The multi-modal RNN model comprises a radar module and a meteorological element module, and the multi-modal RNN model obtains a short-term precipitation prediction result according to a predicted radar map.

[0038] 3. An electronic device comprising a processor and a storage device, wherein the storage device stores a plurality of instructions, and the processor is configured to read the plurality of instructions in the storage device and execute the above method.

[0039] The short-term precipitation prediction method, device and equipment based on the multi-modal RNN provided by the application have at least the following beneficial effects:

[0040] (1) The structure of the MS-RNN model is improved, and the MM-RNN model capable of modal fusion is obtained, the method of using meteorological element and radar two modal information for short-term precipitation prediction is realized, the radar information is mainly used for prediction, the meteorological element information is used as auxiliary, the factors considered are more comprehensive, the prediction accuracy is high, when the MM-MotionRNN is used, the critical success index (CSI) of precipitation greater than 10mm / h increases by 23.4%, and the mean square error (MSE) decreases by 6.7%.

[0041] (2) The meteorological element data is used for auxiliary prediction, the interpolated meteorological element data is relatively continuous and easy to predict, and the problem that the radar precipitation data is difficult to predict due to spatial sparseness can be effectively solved. In addition, the introduction of meteorological element data can force the potential distribution of radar data to comply with a certain meteorological rule instead of being randomly distributed. The change of meteorological elements can slow down the error accumulation in the autoregressive process of the RNN model, which is beneficial to long-term precipitation prediction. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 A flowchart of one embodiment of the short-term precipitation prediction method based on the multi-modal RNN provided by the present application;

[0043] Figure 2 A structural schematic diagram of one embodiment of the MS-RNN model in the multi-modal prediction model used in the short-term precipitation prediction method provided by the present application;

[0044] Figure 3 A schematic diagram of one embodiment of the prediction data flow in the short-term precipitation prediction method provided by the present application;

[0045] Figure 4 A structural schematic diagram of the model for short-term precipitation prediction using a single mode in the prior art;

[0046] Figure 5 A structural schematic diagram of one embodiment of the multi-modal prediction model used in the short-term precipitation prediction method provided by the present application;

[0047] Figure 6 A structural schematic diagram of one embodiment of the mode fusion unit in the multi-modal prediction model used in the short-term precipitation prediction method provided by the present application;

[0048] The reference signs: EE0-first encoder, EE1-second encoder, EE2-third encoder, ED2-first decoder, ED1-second decoder, ED0-third decoder, RE0-fourth encoder, RE1-fifth encoder, RE2-sixth encoder, RD2-fourth decoder, RD1-fifth decoder, RD0-sixth decoder. DETAILED DESCRIPTION

[0049] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings in the specification and specific embodiments.

[0050] Embodiment one:

[0051] Reference Figure 1 In some embodiments, a short-term precipitation prediction method based on a multi-modal RNN is provided, and the method comprises:

[0052] S1, obtaining a radar chart and a meteorological element chart;

[0053] S2, training a multi-modal RNN model by using the radar chart and the meteorological element chart;

[0054] S3, performing precipitation prediction by using the trained multi-modal RNN model.

[0055] The multi-modal RNN model comprises a radar module and a meteorological element module, and the multi-modal RNN model obtains a nowcasting precipitation prediction result according to a predicted radar chart.

[0056] It should be noted that NWP is the basis for today's operational weather forecasting, which mainly takes terrain, ground observation, upper air observation, radar and satellite data as input, and uses physical laws to simulate atmospheric dynamics to predict future precipitation, temperature, pressure and wind and other weather conditions. Although NWP is mainly used for large-scale and long-term weather forecasting, we can still learn its working principle in the short term to improve the accuracy of nowcasting precipitation prediction. Inspired by NWP, the embodiment proposes a new multi-modal model MM-RNN to explore the complex principles behind precipitation evolution. MM-RNN mainly takes radar and ground observation data as input, and can simultaneously perform meteorological element (including wind direction, wind speed, humidity, dew point, temperature, pressure, etc.) prediction and nowcasting precipitation prediction. The meteorological element data in the embodiment is collected by ground observation stations. MM-RNN uses an advanced multi-scale RNN model (MS-RNN) as the basic structure, and introduces a modal fusion unit to filter modal information conducive to precipitation prediction.

[0057] In NWP, radar and satellite data are usually used as assimilation data to correct observation data. However, ground observation data is prone to be missing in the spatial and temporal dimensions; there are few upper air observation stations and the data sampling interval is long; in addition, the resolution and accuracy of satellite data are generally low. Therefore, we mainly use radar data and auxiliary ground station data to predict precipitation. In the training process, the encoder of the single-modal RNN model is poorly organized, and the latent space is not regular enough. The introduction of meteorological element information can force the latent distribution of radar data to follow deterministic meteorological laws to some extent, rather than random distribution, directly learn meteorological laws, and do not have to gradually learn from a large amount of training data, which can effectively solve this problem.

[0058] The MM-RNN model provided in the embodiment introduces meteorological element knowledge into radar precipitation prediction, which is a key part of the entire model.

[0059] Specifically, in step S1, the meteorological element map is subjected to interpolation processing. Because the time interval of the meteorological element data observed by the ground observation station is greater than the time interval of the radar map, and the ground stations are distributed discretely while the radar covers a wide area, the meteorological element data needs to be interpolated to keep consistent with the time and spatial resolution of the radar. The precipitation data in the radar map is sparse in space and has a long-tail feature in overall distribution, which is difficult to use for prediction, while the interpolated meteorological element data is relatively continuous and easy to predict. Therefore, inputting the interpolated meteorological element map into the multi-modal RNN model to assist radar prediction can make the short-impending precipitation prediction better and more accurate.

[0060] In steps S2-S3, the meteorological element module and the radar module are both MS-RNN models (multi-scale RNN models), which include three encoders and three decoders. The encoders use pooling operations, and the decoders use up-sampling operations. The MS-RNN model is described in detail as follows:

[0061] The MS-RNN model (Multi-Scale RNN) uses multi-scale technology to improve the previous RNN models (such as ConvLSTM, TrajGRU, PredRNN, PredRNN++, MIM, and MotionRNN). In short, MS-RNN embeds the UNet structure into the stacked RNN layer, and still uses the Seq2Seq architecture used by conventional RNN. Compared with ordinary RNN models, MS-RNN has lower memory requirements without increasing the number of model parameters, but has stronger spatio-temporal dynamic modeling capability. MS-RNN is a general framework that can easily transplant previous RNN models without changing their internal structure. In this embodiment, MS-RNN is used as the basic module of the MM-RNN model (multi-modal RNN model), so the above-mentioned MM-RNN model can also be compatible with multiple RNN models like MS-RNN, forming MM-ConvLSTM, MM-TrajGRU, MM-PredRNN, MM-PredRNN++, MM-MIM, and MM-MotionRNN models.

[0062] Preferably, this embodiment uses a more advanced multi-scale motion RNN model (MS-MotionRNN model) as a representative model to describe the MM-RNN model. Referring to Figure 2 which shows the structure and data propagation path of the MS-MotionRNN model. Compared with the MotionRNN model, the structure of the MS-MotionRNN increases additional skip connections, data down-sampling, and up-sampling operations. Similar to the MS-RNN model, part of the data and propagation path of the MM-RNN model is optional, which depends on the basic model used.

[0063] Referring to Figure 3 In the MM-RNN model, two input-output prediction flows are included, which are radar flow and meteorological element flow. The prediction process is as follows: the radar chart and the meteorological element chart at the same time are input into the radar module and the meteorological element module respectively to form the radar flow and the meteorological element flow. The radar flow passes through the radar module to output the predicted radar chart, and the meteorological element flow passes through the meteorological element module to output the predicted meteorological element chart.

[0064] It should be noted that there is also a one-way connection between the two prediction flows, that is, the input of the radar module also includes the meteorological element flow from the meteorological element module, and the radar prediction chart output by the radar prediction module is the result of the modal fusion of the meteorological element flow and the radar flow.

[0065] Figure 4 An MS-RNN model using a single radar mode for short-term precipitation prediction is shown, which inputs the real radar chart R t or the predicted radar chart at the current time outputs the predicted radar chart at the next time The model is used to compare with the multi-modal prediction method provided in the embodiment.

[0066] Referring to Figure 5 which shows the specific structure diagram of the multi-modal RNN model provided in the embodiment. Among them, the meteorological element module (EM module) is an MS-RNN model, which is composed of a first encoder EE0, a second encoder EE1, a third encoder EE2, a first decoder ED2, a second decoder ED1 and a third decoder ED0. As shown in the figure, the meteorological element flow is input into the first encoder EE0, and sequentially passes through the first encoder EE0, the second encoder EE1, the third encoder EE2, the first decoder ED2, the second decoder ED1 and the third decoder ED0. It also includes an additional jump connection, specifically, the meteorological element flow of the first encoder EE0 also flows to the third decoder ED0, and the meteorological element flow of the second encoder EE1 also flows to the second decoder ED1. The third decoder ED0 outputs the predicted meteorological element chart.

[0067] In a one-step prediction process, the input of the meteorological element module is the real meteorological element chart E t or the predicted meteorological element chart at the current time The output is the predicted meteorological element chart at the next time Specifically, the encoder (EE) and the decoder (ED) of EM are both composed of 3 RNN units (EE i and ED i, i = 0, 1, 2). EE uses two max-pooling to obtain multi-scale representation, which reduces the original feature maps by 4 times. The data flow in EE can be represented as:

[0068]

[0069]

[0070]

[0071] where, represents the output data of the first encoder, represents the output data of the second encoder, represents the output data of the third encoder, ↓ represents a down-sampling operation.

[0072] In contrast to EE, ED uses two up-sampling to restore the feature maps to the original size. The data flow in ED can be represented as:

[0073]

[0074]

[0075]

[0076]

[0077] where, represents the output data of the first decoder, represents the output data of the second decoder, represents the output data of the third decoder, ↑ represents an up-sampling operation, represents the hidden state of the RNN output in the meteorological element module, represents the spatio-temporal memory state of the RNN output in the meteorological element module.

[0078] The radar module (RM module) is an MS-RNN model, consisting of a fourth encoder RE0, a fifth encoder RE1, a sixth encoder RE2, a fourth decoder RD2, a fifth decoder RD1, and a sixth decoder RD0. The radar stream is input into the fourth encoder RE0, and then sequentially passes through the fourth encoder RE0, fifth encoder RE1, sixth encoder RE2, fourth decoder RD2, fifth decoder RD1, and sixth decoder RD0. Additional skip connections are also included; specifically, the radar stream from the fourth encoder RE0 also flows to the sixth decoder RD0, and the radar stream from the fifth encoder RE1 also flows to the fifth decoder RD1.

[0079] and Figure 4 Unlike the single-modal prediction model shown, there is a unidirectional connection between the two prediction streams; that is, the input of the radar module also includes the meteorological element stream from the meteorological element module. For example... Figure 5 As shown, specifically: the meteorological element stream of the first decoder ED2 also flows to the fourth decoder RD2, the meteorological element stream of the second decoder ED1 also flows to the fifth decoder RD1, and the meteorological element stream of the third decoder ED0 also flows to the sixth decoder RD0. The meteorological element stream and the radar stream are fused in the fourth decoder RD2, the fifth decoder RD1, and the sixth decoder RD0 through an attention mechanism, and the sixth decoder RD0 finally outputs the predicted radar image.

[0080] In one step of the prediction process, the input to RM is the current real radar image R. t Or the current moment's predicted radar chart The output is the predicted radar chart for the next time step. Specifically, the encoder (RE) and decoder (RD) of an RNN are both composed of three RNN units (RE). i and RD i (i = 0, 1, 2). RE also uses two max pooling operations to obtain multi-scale representations, reducing the original feature map by a factor of 4. The data flow in RE can be represented as:

[0081]

[0082]

[0083]

[0084] in, This represents the output data of the fourth encoder. This represents the output data of the fifth encoder. This represents the output data of the sixth encoder, () ↓ This indicates a downsampling operation.

[0085] The RD model is the most important part of the whole model and is the key to distinguish the multi-modal model MM-RNN from the single-modal model MS-RNN. The RD accepts the output of the ED as a multi-modal input guide and fuses it with the output of the RE and the data inside the RD (if any) through a modal fusion unit (MFM). The data flow in the RD can be expressed as:

[0086]

[0087]

[0088]

[0089]

[0090] wherein, represents the output data of the fourth decoder, represents the output data of the fifth decoder, represents the output data of the sixth decoder, ↑ represents an up-sampling operation, H represents a hidden state output by the RNN in the radar module, and M represents a spatio-temporal memory state output by the RNN in the radar module.

[0091] In some embodiments, the radar module further comprises a modal fusion unit for fusing the meteorological element features and the radar features using an attention mechanism, which is arranged in the fourth decoder RD2, the fifth decoder RD1 and the sixth decoder RD0. The structure of the modal fusion unit is as shown in Figure 6 The MFM uses the radar features as the query Q, and uses the meteorological element features as the key K and the value V, for focusing on the influence of the elements on the movement of the radar precipitation map. The specific formula of the modal fusion unit is as follows:

[0092]

[0093]

[0094] Q = (W q R) T K = W k E, V = W V E,

[0095] S = softmax(QK),

[0096] A = VS T ,

[0097] F = A + R + E

[0098] where R is the radar feature, E is the meteorological element feature, K is the key of the modal fusion unit, V is the value of the modal fusion unit, Q is the query variable of the modal fusion unit, S is the similarity score matrix, A is the attention matrix, W k is the parameter of K, W q is the parameter of Q, W V is the parameter of V, F is the result obtained after modal fusion, ED i (i = 0, 1, 2) represent the decoder in the meteorological element module, RE i and RD i (i = 0, 1, 2) represent the encoder and decoder in the radar module, respectively.

[0099] Since the MM-RNN model is composed of two MS-RNN models, which are used for meteorological element prediction and radar precipitation prediction respectively, we need to punish these two parts to optimize the parameters of the corresponding network. Specifically, both parts use the conventional MSE (Mean Squared Error) + MAE (Mean Absolute Error) loss function, and then they are combined by weighting to obtain the total loss function. The loss function of the multi-modal model is represented as follows:

[0100]

[0101]

[0102] L total = λ1L ele + λ2L rad ,

[0103] where L total is the model loss function, L ele is the meteorological element loss function, L rad is the radar loss function, m + n is the total length of the radar graph sequence or the meteorological element graph sequence, λ1 and λ2 are the meteorological element loss weight and the radar loss weight, respectively, is the predicted radar graph, R t is the real radar graph, is the predicted meteorological element graph, E t is the real meteorological element graph.

[0104] Since the element prediction module needs to predict the values of 7 elements at the same time at each time, it is a relatively difficult task, and we need to give it a larger penalty. However, precipitation prediction is our main task, and this part cannot be ignored. Therefore, we set both hyperparameters λ1 and λ2 to 1 as a trade-off.

[0105] Example two:

[0106] In some embodiments, a short-term precipitation prediction device based on a multi-modal RNN is provided, comprising:

[0107] a picture acquisition module configured to acquire a radar picture and a meteorological element picture;

[0108] a training module configured to train a multi-modal model using the radar picture and the meteorological element picture, the multi-modal prediction model comprising a radar module and a meteorological element module;

[0109] a prediction module configured to perform precipitation prediction using the trained multi-modal prediction model.

[0110] In the picture acquisition module, the meteorological element picture is subjected to interpolation processing.

[0111] In some embodiments, the prediction process of the multi-modal RNN model is as follows:

[0112] the radar picture and the meteorological element picture at the same time are input into the radar module and the meteorological element module respectively to form a radar stream and a meteorological element stream, and the input of the radar module further comprises the meteorological element stream from the meteorological element module;

[0113] the radar module outputs a predicted radar picture, and the meteorological element module outputs a predicted meteorological element picture.

[0114] In some embodiments, the meteorological element module and the radar module are both multi-scale RNN models, comprising three encoders and three decoders, the encoders use pooling operations, and the decoders use up-sampling operations.

[0115] wherein the meteorological element stream sequentially passes through a first encoder, a second encoder, a third encoder, a first decoder, a second decoder, and a third decoder, the meteorological element stream of the first encoder further flows to the third decoder, and the meteorological element stream of the second encoder further flows to the second decoder; and the third decoder outputs a predicted meteorological element picture.

[0116] wherein the radar stream sequentially passes through a fourth encoder, a fifth encoder, a sixth encoder, a fourth decoder, a fifth decoder, and a sixth decoder, the radar stream of the fourth encoder further flows to the sixth decoder, and the radar stream of the fifth encoder further flows to the fifth decoder;

[0117] the meteorological element stream of the first decoder further flows to the fourth decoder, the meteorological element stream of the second decoder further flows to the fifth decoder, and the meteorological element stream of the third decoder further flows to the sixth decoder; and the sixth decoder outputs a predicted radar picture.

[0118] In some embodiments, the radar module further comprises a modal fusion unit for fusing the weather element features and the radar features by using an attention mechanism.

[0119] In some embodiments, the modal fusion unit calculates the formula as follows:

[0120]

[0121]

[0122] Q=(W q R) T ,K=W k E,V=W V E,

[0123] S=softmax(QK),

[0124] A=VS T ,

[0125] F=A+R+E

[0126] Wherein, R is the radar feature, E is the weather element feature, K is the key of the modal fusion unit, V is the value of the modal fusion unit, Q is the query variable of the modal fusion unit, S is the similarity score matrix, A is the attention matrix, W k , W q , W V are the parameters of K, Q and V respectively, F is the result obtained after modal fusion, ED i (i=0,1,2) represent the decoder in the weather element module, RE i and RD i (i=0,1,2) represent the encoder and the decoder in the radar module respectively.

[0127] In some embodiments, the loss function of the multi-modal RNN model is represented as follows:

[0128]

[0129]

[0130] L total =λ1L ele +λ2L rad ,

[0131] Wherein, L total is the model loss function, L ele is the weather element loss function, L rad is the radar loss function, m+n is the total length of the radar image sequence or the weather element image sequence, λ1 and λ2 are the weather element loss weight and the radar loss weight respectively, for predicting a radar map, R t for a true radar map, for predicting a weather element map, E t for a true weather element map.

[0132] Embodiment Three:

[0133] In some embodiments, an electronic device is provided, comprising a processor and a storage device, the storage device having a plurality of instructions stored therein, the processor being configured to read the plurality of instructions in the storage device and perform the above method.

[0134] The beneficial effects of the present application are further illustrated below through comparative test experiments.

[0135] We conducted experiments on two multi-modal datasets (MeteoNet and RAIN-F) using the MM-RNN model and the single-modal model MS-RNN, and the results showed that the MM-RNN model was superior to the MS-RNN model using a single radar modality. On the MeteoNet dataset, compared with the MS-MotionRNN model, when using the MM-MotionRNN, the critical success index (CSI index) of the precipitation R greater than 10 mm / h increased by 23.4%, and the mean square error (MSE index) decreased by 6.7%; on the RAIN-F dataset, compared with the MS-MIM model, when using the MM-MIM model, the Heidke skill score (HSS index) of the precipitation R greater than 5 mm / h increased by 209.4%, and the weighted mean square error (B-MSE index) decreased by 4.6%. From the above experimental results, it can be known that using the multi-modal RNN model provided in the embodiment for multi-modal fusion short-term precipitation prediction can significantly improve the prediction accuracy.

[0136] The short-term precipitation prediction method, device and equipment based on the multi-modal RNN provided by the embodiment improve the structure of the MS-RNN model, obtain the MM-RNN model capable of modal fusion, realize the short-term precipitation prediction by the method of fusing the two modes of meteorological element information and radar information, mainly use the radar information, supplementarily use the meteorological element information, comprehensively consider factors, have high prediction accuracy, when the MM-MotionRNN is used, the critical success index (CSI) of precipitation greater than 10mm / h is increased by 23.4%, and the mean square error (MSE) is reduced by 6.7%; the meteorological element data is used for auxiliary prediction, the interpolated meteorological element data is relatively continuous and easy to predict, and the problem that the radar precipitation data is sparse in space and difficult to predict can be effectively solved; and the introduction of the meteorological element data can force the potential distribution of the radar data to comply with the deterministic meteorological law instead of random distribution to a certain extent; the change of the meteorological element can slow down the error accumulation in the self-recurrence process of the RNN model, and is beneficial to long-term precipitation prediction.

[0137] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to encompass within their scope all such variations and modifications as are included within the scope of the application. It should be apparent that the application is not limited to the specific embodiments described herein, but can be practiced with modification and alteration within the scope and spirit of the present application. Accordingly, the specification is to be regarded in an illustrative, rather than a restrictive sense, as the application is entitled to protection based on the full scope and range of equivalents of the claims, including all modifications and variations that can be made thereto.

Claims

1. A multi-modal RNN based nowcasting precipitation prediction method, characterized in that, The method comprises: acquiring a radar chart and a meteorological element chart; training a multi-modal RNN model using the radar chart and the meteorological element chart; performing precipitation prediction using the trained multi-modal RNN model; wherein the multi-modal RNN model comprises a radar module and a meteorological element module, and the multi-modal RNN model obtains a short-term precipitation prediction result according to a predicted radar chart; the meteorological element module and the radar module are both multi-scale RNN models, comprising three encoders and three decoders, the encoders use a pooling operation, and the decoders use an up-sampling operation; the meteorological elements sequentially pass through a first encoder, a second encoder, a third encoder, a first decoder, a second decoder, and a third decoder, the meteorological element stream of the first encoder also flows to the third decoder, and the meteorological element stream of the second encoder also flows to the second decoder; the third decoder outputs a predicted meteorological element chart; a radar stream sequentially passes through a fourth encoder, a fifth encoder, a sixth encoder, a fourth decoder, a fifth decoder, and a sixth decoder, the radar stream of the fourth encoder also flows to the sixth decoder, and the radar stream of the fifth encoder also flows to the fifth decoder; the meteorological element stream of the first decoder also flows to the fourth decoder, the meteorological element stream of the second decoder also flows to the fifth decoder, and the meteorological element stream of the third decoder also flows to the sixth decoder; the sixth decoder outputs a predicted radar chart; the radar module further comprises a modal fusion unit for fusing meteorological element features and radar features using an attention mechanism; the calculation formula of the modal fusion unit is as follows: wherein R is a radar feature, E is a meteorological element feature, K is a key of the modal fusion unit, V is a value of the modal fusion unit, Q is a query variable of the modal fusion unit, S is a similarity score matrix, A is an attention matrix, , , are parameters of K, Q, V respectively, F is a result obtained after modal fusion, ED i represents a decoder in the meteorological element module, RE i and RD i respectively represent an encoder and a decoder in the radar module, wherein i = 0, 1, 2.

2. The method of claim 1, wherein, the prediction process of the multi-modal RNN model is as follows: input the radar chart and the meteorological element chart at the same time into the radar module and the meteorological element module respectively to form a radar stream and a meteorological element stream, and the input of the radar module further comprises the meteorological element stream from the meteorological element module; the radar module outputs a predicted radar chart, and the meteorological element module outputs a predicted meteorological element chart.

3. The method of claim 1, wherein, the loss function of the multi-modal RNN model is represented as follows: wherein, is a model loss function, is a meteorological element loss function, is a radar loss function, is a total length of the sequence of radar maps or the sequence of meteorological element maps, and are a meteorological element loss weight and a radar loss weight, respectively, is a predicted radar map, is a true radar map, is a predicted meteorological element map, is a true meteorological element map.

4. A short-term precipitation forecasting device based on a multi-modal RNN, characterized by, comprises: a picture acquisition module for acquiring a radar chart and a meteorological element chart; a training module for training a multi-modal RNN model using the radar chart and the meteorological element chart; a prediction module for performing precipitation prediction using the trained multi-modal RNN model; wherein the multi-modal RNN model comprises a radar module and a meteorological element module, and the multi-modal RNN model obtains a short-term precipitation prediction result according to a predicted radar chart; the meteorological element module and the radar module are both multi-scale RNN models, comprising three encoders and three decoders, the encoders use a pooling operation, and the decoders use an up-sampling operation; the meteorological elements sequentially pass through a first encoder, a second encoder, a third encoder, a first decoder, a second decoder, and a third decoder, the meteorological element stream of the first encoder also flows to the third decoder, and the meteorological element stream of the second encoder also flows to the second decoder; the third decoder outputs a predicted meteorological element chart; a radar stream sequentially passes through a fourth encoder, a fifth encoder, a sixth encoder, a fourth decoder, a fifth decoder, and a sixth decoder, the radar stream of the fourth encoder also flows to the sixth decoder, and the radar stream of the fifth encoder also flows to the fifth decoder; the meteorological element stream of the first decoder also flows to the fourth decoder, the meteorological element stream of the second decoder also flows to the fifth decoder, and the meteorological element stream of the third decoder also flows to the sixth decoder; the sixth decoder outputs a predicted radar chart; The radar stream sequentially passes through a fourth encoder, a fifth encoder, a sixth encoder, a fourth decoder, a fifth decoder, and a sixth decoder, the radar stream of the fourth encoder also flows to the sixth decoder, and the radar stream of the fifth encoder also flows to the fifth decoder; The meteorological element stream of the first decoder also flows to the fourth decoder, the meteorological element stream of the second decoder also flows to the fifth decoder, and the meteorological element stream of the third decoder also flows to the sixth decoder; and the sixth decoder outputs a predicted radar map; The radar module further comprises a modal fusion unit, configured to fuse meteorological element features and radar features by using an attention mechanism; The modal fusion unit is calculated according to the following formula: wherein R is a radar feature, E is a meteorological element feature, K is a key of the modal fusion unit, V is a value of the modal fusion unit, Q is a query variable of the modal fusion unit, S is a similarity score matrix, A is an attention matrix, , , are parameters of K, Q, V respectively, F is a result obtained after modal fusion, ED i represents a decoder in the meteorological element module, RE i and RD i respectively represent an encoder and a decoder in the radar module, wherein i = 0, 1, 2.

5. An electronic device comprising a processor and a storage device, characterized in that, The storage device stores a plurality of instructions, and the processor is configured to read the plurality of instructions in the storage device and execute the method according to any one of claims 1-3.

Images