A radar plot extrapolation prediction method, system, device, and medium

Abstract

This invention discloses a radar image extrapolation prediction method, system, device, and medium, relating to the field of meteorological forecasting technology. The method includes: acquiring the current real radar image and the previous real radar image; inputting the current real radar image and the previous real radar image into a radar image extrapolation prediction model for prediction, obtaining a predicted radar image for the next moment; the predicted radar image for the next moment is used to predict cloud distribution and precipitation conditions for the next moment; the radar image extrapolation prediction model is determined based on an optical flow extraction network and a spatiotemporal convolutional network. This invention can improve prediction accuracy and obtain clearer and more accurate radar images.

Description

Technical Field

[0001] This invention relates to the field of meteorological forecasting technology, and in particular to a radar image extrapolation forecasting method, system, device, and medium. Background Technology

[0002] Precipitation is a meteorological manifestation of a dynamic, nonlinear, multi-timescale circulation system, and also a product of the combination of local circulation, thermal effects, and topography. China is a vast country with complex topography and significant climate differences across regions. Especially in the south, located in the monsoon region, very heavy short-duration rainfall often occurs in summer, triggering urban flooding and natural disasters such as flash floods and mudslides. Therefore, providing accurate and rapid precipitation forecasts is crucial, not only for water resource development and utilization but also for the safety of everyone's life and property. Radar can sensitively detect rainfall; the radar reflectivity factor of rainfall is generally between 20 and 50 dBZ, and the larger the radar echo, the greater the corresponding precipitation intensity. Therefore, a widely used method for precipitation forecasting is to first predict future radar data using historical radar data, and then use the predicted radar data to generate precipitation forecasts. The key lies in accurately predicting radar data at a specific future moment. Currently, optical flow methods and time series forecasting methods are mainly used.

[0003] Optical flow, as the name suggests, is the flow of light. For example, a shooting star streaking across the night sky as perceived by the human eye. In computer vision, it defines the movement of an object in an image, which can be caused by camera movement or object movement. Specifically, it refers to the amount of movement of a pixel representing the same object in one frame of a video image to the next frame, represented by a two-dimensional vector. One variable represents the horizontal movement of the point, and the other represents the vertical movement. The optical flow field is a two-dimensional vector field that reflects the trend of grayscale changes at each point in the image. It can be seen as the instantaneous velocity field generated by the movement of grayscale pixels on the image plane. It contains the instantaneous velocity vector information of each pixel. The purpose of studying the optical flow field is to approximate the motion field, which cannot be directly obtained, from a sequence of images. Ideally, the optical flow field corresponds to the motion field.

[0004] Solving the optical flow field mainly relies on two fundamental assumptions: First, brightness remains constant. That is, the brightness of the same target does not change when it moves between different frames. This is an assumption of the basic optical flow method (all variants of optical flow methods must satisfy it) and is used to derive the basic equations of the optical flow method. Second, time is continuous or the motion is "small motion". That is, changes in time will not cause drastic changes in the target's position, and the displacement between adjacent frames must be relatively small. This is also an indispensable assumption of the optical flow method. Through the above assumptions and derivations, the following formulas can be obtained:

[0005] I xu+I y v+I t =0

[0006] Among them, I x I y I t Let represent the partial derivatives of the grayscale value of a pixel in the image along the X, Y, and T directions (T refers to the time direction, i.e., the change in pixel value at the same position in the image between two consecutive moments), which can be obtained from the image data. u and v are the velocity vectors of optical flow along the X and Y axes, respectively, which are the desired values. There is only one equation, but two unknowns, making it impossible to obtain the exact values ​​of u and v. Therefore, additional constraints need to be introduced. Introducing constraints from different perspectives leads to different methods for calculating optical flow fields. Based on the differences in theoretical foundation and mathematical methods, they can be divided into four types: gradient-based (differential) methods, matching-based methods, energy-based (frequency) methods, phase-based methods, and neurodynamic methods.

[0007] Besides distinguishing optical flow methods based on their underlying principles, they can also be categorized into dense and sparse optical flow based on the density of the two-dimensional vectors within the resulting optical flow field. Dense optical flow is an image registration method that performs point-by-point matching on an image or a specified region. It calculates the offset of all points in the image, thus forming a dense optical flow field. Pixel-level image registration can then be performed using this dense optical flow field. In contrast, sparse optical flow does not perform point-by-point calculations for every pixel of the image. It typically requires a set of points to be tracked, ideally possessing certain distinct characteristics, such as Harris corners, which leads to relatively stable and reliable tracking. The computational cost of sparse tracking is significantly lower than that of dense tracking.

[0008] Extrapolating radar echoes using optical flow primarily utilizes the relative motion between two radar image frames during observation to predict future radar images. Optical flow refers to the instantaneous velocity of pixels moving on the observation imaging plane from a moving object in space. It uses the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the relationship between the previous and current frames. Generally, optical flow is generated by the movement of foreground objects in the scene, camera movement, or the combined movement of both. In practical applications, optical flow is usually calculated from historical radar echoes, and then the optical flow field is used to predict future changes, thereby predicting future radar echo images. There are many methods for calculating optical flow, with the Horn-Schunck algorithm and the Lucas-Kanade algorithm being among the most classic.

[0009] like Figure 1As shown, the optical flow method first inputs two radar images, img_0 and img_1, taken at consecutive time points. Then, the corresponding optical flow field is calculated using the optical flow algorithm. Assuming img_0 is the previous image and img_1 is the current image, the optical flow information is obtained by calculating the movement of each point when img_0 changes to img_1. For example, if a point's coordinates in img_0 are (0, 0) and its coordinates in img_1 are (1, 1), then the optical flow information for this point is (1, 1). The optical flow information is then used to predict the radar image at the next time point, thus completing the prediction.

[0010] Using optical flow to predict future images has limitations. Since optical flow represents changes in objects between two frames, it cannot utilize longer-term historical information. Therefore, existing optical flow-based algorithms generally have poor prediction accuracy over longer periods, typically lower than those using convolutional models like ConvLSTM and its improved temporal convolutional models. Furthermore, optical flow methods perform poorly in handling boundary conditions. For example, if the entire image is shifted to the left, predicting the next frame using optical flow alone will result in poor performance on the left side. Additionally, traditional optical flow algorithms, based on formula derivation, are slow despite their wide applicability.

[0011] Besides using mathematical methods to extrapolate optical flow to radar echo maps, deep learning is now popularly used to predict future radar images. The idea is to define precipitation nowcasting as a spatiotemporal sequence prediction problem, where both the input and the target are spatiotemporal sequences (radar echo maps). Thanks to the invention of the ConvLSTM model, it's possible to effectively extract temporal information using the LSTM structure while simultaneously using convolutional neural networks to extract spatial features. Many subsequent works have improved upon ConvLSTM, such as attempting to enable the model to "remember" more historical information or adding auxiliary modules to help the model predict motion changes.

[0012] The earliest method to use deep learning CNNs to solve the optical flow estimation problem was FlowNet, proposed at ICCV 2015, and the same team proposed an improved version, FlowNet 2.0, in 2017, which is considered the most classic paper in the field of optical flow estimation. Figure 2As shown, FlowNet's approach is to use a deep learning end-to-end network model to solve the optical flow estimation problem. Since the ground truth of dense optical flow is the optical flow value of each pixel in the image, manually labeling the optical flow values ​​is almost impossible. Therefore, the authors generate relevant images and the ground truth to be predicted by simulating motion. The improvements in FlowNet 2.0 are mainly reflected in two aspects: firstly, by stacking multiple FlowNet networks, a coarse-to-fine effect is achieved; secondly, FlowNet-SD (Small-Displacement) is designed to be suitable for small offsets by modifying the size of the convolutional kernels and strides in FlowNet to make it more suitable for small offsets.

[0013] Predicting future data of spatiotemporal systems based on historical observational data, i.e., spatiotemporal sequence forecasting (STSF), is an important and challenging problem. While many real-world problems can be viewed as spatiotemporal sequence forecasting problems, and much recent work has utilized machine learning to address this issue.

[0014] In the problem of precipitation prediction using radar echo maps, most existing models employ a combination of recurrent neural networks and convolutional modules to process spatiotemporal data. Long Short-Term Memory (LSTM) networks, due to their unique design and the introduction of a "gating" mechanism, can effectively handle long-term dependencies in sequential data. However, since they were initially designed primarily for natural language processing, they are not suitable for processing time series data with spatial information. To address the issue of LSM networks' inability to adapt to spatial sequence data, Shi Xingjian designed the ConvLSTM model structure, such as... Figure 3 As shown.

[0015] The specific calculation formula for the ConvLSTM model is as follows:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] Where * represents a convolution operation, This represents a product operation. All W and b are parameters of the neural network at that location. For example... Figure 3 bottom left corner Wx and x t Performing convolution operations*, in fact W x W xi W xf W xo W xc It consists of multiple parts, representing all things related to x. t The set of parameters for the neural network performing the convolution operation. Similarly, W x Represents all with h t-1 The set of parameters for a neural network performing convolution operations, including W hi W hf W ho W hc W c Represents all related to c t-1 The set of parameters for a neural network performing convolution operations, including W ci W cf W co b contains b i b f b o b c In general, with W xi For example, W represents the parameters and weights of the neural network, the subscript x indicates that it is used in the operation with variable x, and i indicates that it is used to calculate variable i. Taking the first formula as an example, W... xi W hi W ci These are all model parameters, calculated separately with the variables x, h, and c, with the aim of obtaining the output i; b i This represents the bias used to calculate the output i. i Represents the current input, while h t-1 and c t-1 It is the output of the neural network at the previous time step, h. t and c t This represents the output of the neural network at the current moment. tanh and σ are the activation functions for the two types of neural networks, while i, f, and o are intermediate values ​​in the neural network computation. Compared to the well-known Long Short-Term Memory (LSTM) network, ConvLSTM primarily transforms some product operations into convolution operations, thereby ensuring that spatial information is not lost while extracting temporal information from the data.

[0022] Subsequent experiments revealed that when multiple ConvLSTMs are stacked, the layers become independent of each other, and the bottom layer ignores the time information of the highest layer in the previous time step; furthermore, layers can only communicate with each other via h. t It transmits time information while ignoring the hidden information in c. tThe information in the text. To solve this problem, the improved structure of ConvLSTM, ST-LSTM, was ultimately chosen, specifically for example... Figure 4 and Figure 5 As shown. Figure 4 This is a diagram of the four-layer stacked structure of ST-LSTM. Figure 5 This is a diagram of a four-layer stack of ConvLSTM. Figure 5 The network at each layer mainly uses Figure 3 The substructure of, in which and correspond Figure 3 Output c t and h t That is, the cell state and hidden state of LSTM. and The subscripts represent time t and the l-th layer, respectively. X t The input represents the image at time t, while This represents the model's output prediction image for time t+1. l This represents the weights of the neural network at layer l. Figure 4 Each layer of the network primarily uses ST-LSTM, compared to... Figure 5 In the middle, mainly between layers, there are multiple The calculation method for transmitting information can be found in the formula in the next paragraph. The main improvement of ST-LSTM is the increase in information flow, specifically, the increase in information flow between different layers at the same time, and the increase in information transfer between different time steps.

[0023] The specific calculation formula for the ST-LSTM model is as follows, and it is similar to the formula for calculating ConvLSTM. * represents a convolution operation. This represents a product operation. All W and b are parameters of the neural network at that location. xg For example, W represents the parameters and weights of the neural network, the subscript x indicates that it is used in the operation with variable x, and g indicates that it is used to calculate variable g. t Represents the current image input, while and That is, the output of the neural network at the previous time step. This represents the output of the previous layer of the neural network. If it's the first layer, then... Let's replace it with M from the last layer of the previous moment. and This represents the output of the neural network at the current moment. tanh and σ are the activation functions for the two types of neural networks, while g, i, f, g', i', f', and o are intermediate values ​​in the neural network calculation. Compared to ConvLSTM, the main difference is the addition of the M variable in the output.

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] While the prediction results based on the temporal convolution model are relatively accurate, they are often blurrier than images generated by the optical flow method, which does not meet the needs of practical applications. Furthermore, the model training does not directly constrain the "motion" in the radar image; it can only indirectly constrain it by calculating the loss value between the predicted and actual radar images.

[0035] In summary, existing radar image extrapolation prediction methods suffer from low prediction accuracy and the inability to obtain clear and accurate radar images. Summary of the Invention

[0036] The purpose of this invention is to provide a radar image extrapolation prediction method, system, device, and medium to improve prediction accuracy and obtain clearer and more accurate radar images.

[0037] To achieve the above objectives, the present invention provides the following solution:

[0038] A radar image extrapolation prediction method includes:

[0039] Obtain the current real radar image and the previous real radar image;

[0040] The current and previous real radar images are input into the radar image extrapolation prediction model to obtain the predicted radar image for the next moment. The predicted radar image for the next moment is used to predict the cloud distribution and precipitation conditions for the next moment. The radar image extrapolation prediction model is determined based on an optical flow extraction network and a spatiotemporal convolutional network.

[0041] Optionally, the radar image extrapolation prediction model includes: an optical flow extraction module, a spatiotemporal convolution module, and a prediction module; the current real radar image and the previous real radar image are input into the radar image extrapolation prediction model for prediction to obtain the predicted radar image for the next moment, specifically including:

[0042] The current real radar image and the previous real radar image are input into the optical flow extraction module to extract optical flow, thereby obtaining the predicted optical flow field between the current time and the previous time.

[0043] The real radar image at the current moment is input into the spatiotemporal convolution module for feature extraction to obtain the spatiotemporal feature map at the current moment;

[0044] The predicted optical flow field between the current time and the previous time and the spatiotemporal feature map of the current time are input into the prediction module for prediction to obtain the predicted radar map of the next time.

[0045] Optionally, the method for determining the radar chart extrapolation prediction model specifically includes:

[0046] Obtain a training dataset; the training dataset includes several real radar images from consecutive historical moments; wherein, any three adjacent real radar images from historical moments are used as a training data group; in a training data group, the real radar image from the previous moment is used as the first radar image, the real radar image from the middle moment is used as the second radar image, and the real radar image from the next moment is used as the third radar image; the first radar image and the second radar image constitute a first data pair; the second radar image and the third radar image constitute a second data pair;

[0047] Construct an initial neural network model; the initial neural network model includes: an optical flow extraction network, a spatiotemporal convolutional network, and a prediction network;

[0048] Based on the first data pair, the optical flow extraction network is trained to obtain the optical flow extraction module of the radar image extrapolation prediction model.

[0049] Based on the first data pair, the second data pair, and the optical flow extraction module, the spatiotemporal convolutional network and the prediction network are trained to obtain the spatiotemporal convolutional module and the prediction module of the radar image extrapolation prediction model.

[0050] Optionally, based on the first data pair, the optical flow extraction network is trained to obtain the optical flow extraction module of the radar image extrapolation prediction model, specifically including:

[0051] An optical flow algorithm is used to determine the actual optical flow field corresponding to the first data pair;

[0052] The first data pair is input into the optical flow extraction network for optical flow extraction to obtain the first predicted optical flow field corresponding to the first data pair.

[0053] A first loss value is determined based on the first data for the corresponding real optical flow field and the first data for the corresponding first predicted optical flow field.

[0054] The optical flow extraction network is trained with the goal of minimizing the first loss value to obtain the optical flow extraction module of the radar image extrapolation prediction model.

[0055] Optionally, based on the first data pair, the second data pair, and the optical flow extraction module, the spatiotemporal convolutional network and the prediction network are trained to obtain the spatiotemporal convolutional module and the prediction module of the radar image extrapolation prediction model, specifically including:

[0056] The first data pair is input into the optical flow extraction module for optical flow extraction to obtain the second predicted optical flow field corresponding to the first data pair;

[0057] The second radar image is input into the spatiotemporal convolutional network for feature extraction to obtain the spatiotemporal feature map corresponding to the second radar image.

[0058] The first data pair is input into the prediction network to perform prediction by inputting the second predicted optical flow field corresponding to the first data pair and the spatiotemporal feature map corresponding to the second radar map to obtain the prediction radar map corresponding to the first data pair.

[0059] The second data pair is input into the optical flow extraction module for optical flow extraction to obtain the second predicted optical flow field corresponding to the second data pair;

[0060] The second radar image and the first data pair are input into the optical flow extraction module to extract optical flow, thereby obtaining the second predicted optical flow field corresponding to the predicted radar image.

[0061] Based on the second data, a second loss value is determined for the corresponding second predicted optical flow field and the second predicted optical flow field corresponding to the predicted radar map;

[0062] Based on the first data, a third loss value is determined for the corresponding predicted radar map and the third radar map;

[0063] The spatiotemporal convolutional network and the prediction network are trained with the goal of minimizing the sum of the second loss value and the third loss value, to obtain the spatiotemporal convolutional module and the prediction module of the radar image extrapolation prediction model.

[0064] Optionally, the optical flow extraction network is PWC-Net; the spatiotemporal convolutional network is PredRNN; and the prediction network is a convolutional neural network.

[0065] Optionally, the optical flow algorithm is any one of the Horn-Schunck algorithm, Lucas-Kanada algorithm, Pyramidal LK algorithm, and DIS algorithm.

[0066] A radar image extrapolation prediction system, comprising:

[0067] The radar image acquisition module is used to acquire the current real radar image and the previous real radar image.

[0068] The radar image prediction module is used to input the current real radar image and the previous real radar image into the radar image extrapolation prediction model for prediction, so as to obtain the predicted radar image for the next moment; the predicted radar image for the next moment is used to predict the cloud distribution and precipitation conditions for the next moment; the radar image extrapolation prediction model is determined based on the optical flow extraction network and the spatiotemporal convolutional network.

[0069] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to enable the electronic device to perform the radar image extrapolation prediction method described above.

[0070] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the radar image extrapolation prediction method described above.

[0071] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0072] The radar image extrapolation prediction method provided by this invention adopts a radar image extrapolation prediction model based on an optical flow extraction network and a spatiotemporal convolutional network. It can fuse the information extracted by the optical flow extraction network and the spatiotemporal convolutional network, so that the model can use more information when predicting future radar images and improve the model prediction accuracy. At the same time, since optical flow is introduced as an additional constraint for model training, the model can focus on the relationship between points, making the output image clearer and more accurate. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 Here is a flowchart of an existing optical flow method;

[0075] Figure 2 A flowchart of existing methods using time series forecasting;

[0076] Figure 3 Here is a diagram of the existing ConvLSTM model;

[0077] Figure 4 A diagram showing the four-layer stacked structure of an existing ST-LSTM model;

[0078] Figure 5 A diagram showing the four-layer stacked structure of an existing ConvLSTM model;

[0079] Figure 6 A flowchart of the radar image extrapolation prediction method provided by the present invention;

[0080] Figure 7 A diagram illustrating the training process of the radar image extrapolation prediction model provided by this invention;

[0081] Figure 8 A diagram illustrating the training process of the optical flow extraction module provided by this invention;

[0082] Figure 9 This diagram illustrates the training process of the spatiotemporal convolution module and prediction module provided by the present invention. Detailed Implementation

[0083] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0084] The purpose of this invention is to provide a radar image extrapolation prediction method, system, device, and medium to improve prediction accuracy and obtain clearer and more accurate radar images.

[0085] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0086] This invention integrates information extracted by optical flow extraction networks and spatiotemporal convolutional networks, enabling the model to utilize more information when predicting future images, thereby improving model accuracy. At the same time, it introduces optical flow as an additional constraint for training, allowing the model to focus on the relationships between points, resulting in more accurate and reasonable output images.

[0087] like Figure 6 As shown, the radar image extrapolation prediction method provided by this invention includes:

[0088] Step S1: Obtain the current real radar image and the previous real radar image.

[0089] Step S2: Input the current real radar image and the previous real radar image into the radar image extrapolation prediction model for prediction to obtain the predicted radar image for the next moment; the predicted radar image for the next moment is used to predict the cloud distribution and precipitation conditions for the next moment; the radar image extrapolation prediction model is determined based on the optical flow extraction network and the spatiotemporal convolutional network.

[0090] Furthermore, the radar image extrapolation prediction model includes: an optical flow extraction module, a spatiotemporal convolution module, and a prediction module. Step S2 specifically includes:

[0091] Step S2.1: Input the current real radar image and the previous real radar image into the optical flow extraction module to extract optical flow and obtain the predicted optical flow field between the current time and the previous time.

[0092] Step S2.2: Input the real radar image at the current moment into the spatiotemporal convolution module for feature extraction to obtain the spatiotemporal feature map at the current moment.

[0093] Step S2.3: Input the predicted optical flow field between the current time and the previous time and the spatiotemporal feature map of the current time into the prediction module for prediction to obtain the predicted radar map of the next time.

[0094] Furthermore, Figure 7 This is a diagram illustrating the training process of the radar image extrapolation prediction model provided by the present invention, where I represents the radar image, I t The radar chart representing time t, I t-1 The radar chart representing time t-1, I t+1 The radar chart representing time t+1, I' t+1 F represents the radar image at time t+1 predicted by the model. t,t-1 This represents the optical flow field (i.e., optical flow information) between the radar images at time t and t-1, obtained using a mathematical optical flow algorithm; F' t,t-1 This represents the optical flow field between the radar images at time t and time t-1 predicted by the model, obtained using the optical flow extraction module (or optical flow extraction network); F t+1,t This represents the optical flow field between the radar images at time t+1 and time t, F' t+1,t This represents the optical flow field between the radar images at time t+1 and time t predicted by the model. Both are obtained using a trained optical flow extraction module. H t H is the spatiotemporal feature map at time t. t-1This is the spatiotemporal feature map at time t-1. loss1 is the first loss value, loss2 is the second loss value, and loss3 is the third loss value.

[0095] Figure 7 The optical flow extraction module, spatiotemporal convolution module, and prediction module are all built using neural networks and are the final parts used for prediction. Before training is complete, these three modules are respectively called the optical flow extraction network, spatiotemporal convolution network, and prediction network. Optical flow algorithms refer to existing mathematical optical flow algorithms, such as the KL (Lucas-Kanada) algorithm, the HS (Horn-Schunck) algorithm, and the DIS (Dense Inverse Search) algorithm. These are pre-designed tools that can be directly used to calculate optical flow information. The loss function uses algorithms such as MAE (Mean Absolute Error) and MSE (Mean-Square Error) to calculate the difference between the true value and the model's predicted value.

[0096] like Figure 7 As shown, the method for determining the radar chart extrapolation prediction model specifically includes:

[0097] Step S3.1: Obtain the training dataset; the training dataset includes several real radar images from consecutive historical moments; wherein, any three adjacent real radar images from historical moments are used as a training data group; in a training data group, the real radar image from the previous moment is used as the first radar image, the real radar image from the middle moment is used as the second radar image, and the real radar image from the next moment is used as the third radar image; the first radar image and the second radar image constitute a first data pair; the second radar image and the third radar image constitute a second data pair.

[0098] Step S3.2: Construct an initial neural network model; the initial neural network model includes: an optical flow extraction network, a spatiotemporal convolutional network, and a prediction network.

[0099] Step S3.3: Based on the first data pair, train the optical flow extraction network to obtain the optical flow extraction module of the radar image extrapolation prediction model.

[0100] Step S3.4: Based on the first data pair, the second data pair, and the optical flow extraction module, train the spatiotemporal convolutional network and the prediction network to obtain the spatiotemporal convolutional module and the prediction module of the radar image extrapolation prediction model.

[0101] In this embodiment, the training dataset is a set of spatiotemporal sequence data, consisting of multiple consecutive two-dimensional images. Like a video, each moment in time represents a two-dimensional image, and multiple moments together depict the motion and changes of objects in the image. Common time series prediction models are often built upon feature extraction from continuous images, most commonly using ConvLSTM and its improved models. These models use convolutional layers in the spatiotemporal convolution module to extract image features and recurrent neural network layers in the spatiotemporal convolution module to extract temporal information, obtaining a spatiotemporal feature map, which is then used for prediction. This invention adds an optical flow extraction module to help the model extract the optical flow information of the input data, that is, the motion information of each pixel in the image, thereby assisting the model's prediction module in making predictions.

[0102] Furthermore, this invention uses a mature optical flow algorithm to help the model obtain the ground truth of the corresponding optical flow information, uses loss1 to train the optical flow extraction module, and uses loss2 to assist the prediction module in better exploring the relationship between points, so that the prediction results are better in detail.

[0103] like Figure 8 As shown, step S3.3 specifically includes:

[0104] 1) Use an optical flow algorithm to determine the real optical flow field corresponding to the first data pair.

[0105] Preferably, the optical flow algorithm is any one of the Horn-Schunck algorithm, Lucas-Kanada algorithm, PyramidalLK algorithm, and DIS algorithm.

[0106] 2) Input the first data pair into the optical flow extraction network to extract optical flow and obtain the first predicted optical flow field corresponding to the first data pair.

[0107] 3) Determine the first loss value based on the first data pair corresponding to the actual optical flow field and the first data pair corresponding to the first predicted optical flow field.

[0108] 4) Train the optical flow extraction network with the goal of minimizing the first loss value to obtain the optical flow extraction module of the radar image extrapolation prediction model.

[0109] In practical applications, optical flow algorithms are used to calculate the optical flow information F of radar images at adjacent time points. t,t-1 The value is used as the label, i.e., the true value, for training the optical flow extraction network built from a neural network. The mature DIS algorithm is used. Through training, the neural network predicts F'... t,t-1 Approaching F t,t-1This completes the transfer of knowledge from the optical flow algorithm to the optical flow extraction network, resulting in the trained optical flow extraction module. The preferred optical flow extraction network here is PWC-Net (PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume). After this training step, the parameters of the optical flow extraction module will remain unchanged.

[0110] like Figure 9 As shown, step S3.4 specifically includes:

[0111] 1) Input the first data pair into the optical flow extraction module to extract optical flow and obtain the second predicted optical flow field corresponding to the first data pair.

[0112] 2) Input the second radar image into the spatiotemporal convolutional network for feature extraction to obtain the spatiotemporal feature map corresponding to the second radar image.

[0113] 3) Input the second predicted optical flow field corresponding to the first data pair and the spatiotemporal feature map corresponding to the second radar map into the prediction network for prediction to obtain the predicted radar map corresponding to the first data pair.

[0114] 4) Input the second data pair into the optical flow extraction module to extract optical flow and obtain the second predicted optical flow field corresponding to the second data pair.

[0115] 5) Input the second radar image and the predicted radar image corresponding to the first data pair into the optical flow extraction module to extract optical flow and obtain the second predicted optical flow field corresponding to the predicted radar image.

[0116] 6) Determine the second loss value based on the second data for the corresponding second predicted optical flow field and the second predicted optical flow field corresponding to the predicted radar map.

[0117] 7) Determine the third loss value based on the first data for the corresponding predicted radar map and the third radar map.

[0118] 8) Train the spatiotemporal convolutional network and the prediction network with the goal of minimizing the sum of the second loss value and the third loss value to obtain the spatiotemporal convolutional module and the prediction module of the radar image extrapolation prediction model.

[0119] In practical applications, the optical flow extraction module trained by the aforementioned steps does not participate in gradient descent in this step. That is, the neural network parameters remain unchanged, and the spatiotemporal convolutional network and prediction network are mainly trained to obtain the spatiotemporal convolutional module and prediction module.

[0120] Among them, the spatiotemporal convolutional network is preferably PredRNN (Predictive Recurrent Neural Network), which is built based on convolutional LSTM. The spatiotemporal convolutional network processes and analyzes the time series data, outputting a spatiotemporal feature map H containing both temporal and spatial information. t (i.e., spatiotemporal convolutional network based on the current input I) t Spatiotemporal feature map H from the previous moment t-1 The high-dimensional features of the predicted output are similar to the hidden state in a recurrent neural network, and are a type of feature map.

[0121] The prediction network is a convolutional neural network, specifically comprising a first convolutional layer, a first normalization layer, a first activation function layer, a second convolutional layer, a second normalization layer, a second activation function layer, and a third convolutional layer connected in sequence. Its input is F' at the current time step. t,t-1 and H t The output is the predicted I' for the next time step. t+1 By combining the spatiotemporal feature map H containing temporal and spatial information with the optical flow information F, the radar image I' at the next moment is predicted. t+1 .

[0122] For the spatiotemporal convolutional network and the prediction network, the spatiotemporal convolutional module and the prediction module are trained by combining two loss functions. One loss function, I', is used to compare the outputs. t+1 And the real I t+1 The difference between them, as loss3, is another one used to compare I' t+1 with I t Optical flow F' between t+1,t and I t+1 with I t Optical flow F between t+1,t The difference is represented by loss2.

[0123] Since the model trained in the first step can extract optical flow information through the optical flow extraction module, and optical flow information contains motion information, the task of the spatiotemporal convolution module becomes relatively lighter, allowing it to focus more on spatiotemporal information beyond optical flow during the learning process. Traditional model training, however, only uses the predicted I' t+1 And the real I t+1This invention uses optical flow loss to train the model by considering the differences between points, allowing the model to focus on the relationships between points and resulting in more accurate generated images.

[0124] Ultimately, the trained optical flow extraction module, spatiotemporal convolution module, and prediction module together constitute the radar image extrapolation prediction model. This model is generated by inputting two radar images: the current image and the one from the previous time step. t and I t-1 This will output the predicted radar image I' for the next moment. t+1 .

[0125] To implement the above method and achieve the corresponding functions and technical effects, a radar image extrapolation prediction system is provided below. This system includes a radar image acquisition module and a radar image prediction module. The radar image acquisition module acquires the current and previous real radar images. The radar image prediction module inputs the current and previous real radar images into a radar image extrapolation prediction model to obtain a predicted radar image for the next moment. The predicted radar image for the next moment is used to predict cloud distribution and precipitation conditions. The radar image extrapolation prediction model is determined based on an optical flow extraction network and a spatiotemporal convolutional network.

[0126] Furthermore, the present invention also provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the radar image extrapolation prediction method described above. The electronic device may be a server.

[0127] In addition, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described radar image extrapolation prediction method.

[0128] The radar image extrapolation prediction model provided by this invention combines optical flow information and spatiotemporal information. It divides the feature extraction part of the model into two components, one for extracting optical flow information and the other for extracting spatiotemporal information. This allows the model to utilize more information during prediction, resulting in better final prediction results. Furthermore, this invention utilizes optical flow information to assist model training and convergence, enabling the model to focus more on the correspondence between points in preceding and subsequent radar images during training, thereby improving model accuracy.

[0129] Compared with the prior art, the present invention has the following advantages:

[0130] 1. The information extracted by the optical flow extraction module and the spatiotemporal convolution module is fused together. By imposing predictive constraints on the optical flow extraction module, the motion information in the radar image is emphasized.

[0131] 2. By combining the optical flow extraction module and the spatiotemporal convolution module, the model's performance in detail contours is improved, making the model's prediction results closer to actual radar images.

[0132] 3. The model framework is highly scalable; the optical flow extraction module and the spatiotemporal convolution module can be switched to other model structures.

[0133] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0134] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A radar image extrapolation prediction method, characterized in that, include: Obtain the current real radar image and the previous real radar image; The current real radar image and the previous real radar image are input into the radar image extrapolation prediction model to make a prediction and obtain the predicted radar image for the next moment. The radar image extrapolation prediction model includes: an optical flow extraction module, a spatiotemporal convolution module, and a prediction module; The current and previous radar images are input into the radar image extrapolation prediction model to obtain the predicted radar image for the next time step. Specifically, this includes: The current real radar image and the previous real radar image are input into the optical flow extraction module to extract optical flow, thereby obtaining the predicted optical flow field between the current time and the previous time. The real radar image at the current moment is input into the spatiotemporal convolution module for feature extraction to obtain the spatiotemporal feature map at the current moment; The predicted optical flow field between the current time and the previous time and the spatiotemporal feature map of the current time are input into the prediction module for prediction to obtain the predicted radar map of the next time. The predicted radar image for the next moment is used to predict the cloud distribution and precipitation conditions for the next moment; the radar image extrapolation prediction model is determined based on the optical flow extraction network and the spatiotemporal convolutional network.

2. The radar image extrapolation prediction method according to claim 1, characterized in that, The method for determining the radar chart extrapolation prediction model specifically includes: Obtain a training dataset; the training dataset includes several real radar images from consecutive historical moments; wherein, any three adjacent real radar images from historical moments are used as a training data group; in a training data group, the real radar image from the previous moment is used as the first radar image, the real radar image from the middle moment is used as the second radar image, and the real radar image from the next moment is used as the third radar image; the first radar image and the second radar image constitute a first data pair; the second radar image and the third radar image constitute a second data pair; Construct an initial neural network model; the initial neural network model includes: an optical flow extraction network, a spatiotemporal convolutional network, and a prediction network; Based on the first data pair, the optical flow extraction network is trained to obtain the optical flow extraction module of the radar image extrapolation prediction model. Based on the first data pair, the second data pair, and the optical flow extraction module, the spatiotemporal convolutional network and the prediction network are trained to obtain the spatiotemporal convolutional module and the prediction module of the radar image extrapolation prediction model.

3. The radar image extrapolation prediction method according to claim 2, characterized in that, Based on the first data pair, the optical flow extraction network is trained to obtain the optical flow extraction module of the radar image extrapolation prediction model, which specifically includes: An optical flow algorithm is used to determine the actual optical flow field corresponding to the first data pair; The first data pair is input into the optical flow extraction network for optical flow extraction to obtain the first predicted optical flow field corresponding to the first data pair. A first loss value is determined based on the first data for the corresponding real optical flow field and the first data for the corresponding first predicted optical flow field. The optical flow extraction network is trained with the goal of minimizing the first loss value to obtain the optical flow extraction module of the radar image extrapolation prediction model.

4. The radar image extrapolation prediction method according to claim 2, characterized in that, Based on the first data pair, the second data pair, and the optical flow extraction module, the spatiotemporal convolutional network and the prediction network are trained to obtain the spatiotemporal convolutional module and the prediction module of the radar image extrapolation prediction model, specifically including: The first data pair is input into the optical flow extraction module for optical flow extraction to obtain the second predicted optical flow field corresponding to the first data pair; The second radar image is input into the spatiotemporal convolutional network for feature extraction to obtain the spatiotemporal feature map corresponding to the second radar image. The first data pair is input into the prediction network to perform prediction by inputting the second predicted optical flow field corresponding to the first data pair and the spatiotemporal feature map corresponding to the second radar map to obtain the prediction radar map corresponding to the first data pair. The second data pair is input into the optical flow extraction module for optical flow extraction to obtain the second predicted optical flow field corresponding to the second data pair; The second radar image and the first data pair are input into the optical flow extraction module to extract optical flow, thereby obtaining the second predicted optical flow field corresponding to the predicted radar image. Based on the second data, a second loss value is determined for the corresponding second predicted optical flow field and the second predicted optical flow field corresponding to the predicted radar map; Based on the first data, a third loss value is determined for the corresponding predicted radar map and the third radar map; The spatiotemporal convolutional network and the prediction network are trained with the goal of minimizing the sum of the second loss value and the third loss value, to obtain the spatiotemporal convolutional module and the prediction module of the radar image extrapolation prediction model.

5. The radar image extrapolation prediction method according to claim 2, characterized in that, The optical flow extraction network is PWC-Net; the spatiotemporal convolutional network is PredRNN; and the prediction network is a convolutional neural network.

6. The radar image extrapolation prediction method according to claim 3, characterized in that, The optical flow algorithm is any one of the Horn-Schunck algorithm, Lucas-Kanada algorithm, Pyramidal LK algorithm, and DIS algorithm.

7. A radar image extrapolation prediction system, characterized in that, include: The radar image acquisition module is used to acquire the current real radar image and the previous real radar image. The radar image prediction module is used to input the current real radar image and the previous real radar image into the radar image extrapolation prediction model to make a prediction and obtain the predicted radar image for the next moment. The radar image extrapolation prediction model includes: an optical flow extraction module, a spatiotemporal convolution module, and a prediction module; The current and previous radar images are input into the radar image extrapolation prediction model to obtain the predicted radar image for the next time step. Specifically, this includes: The current real radar image and the previous real radar image are input into the optical flow extraction module to extract optical flow, thereby obtaining the predicted optical flow field between the current time and the previous time. The real radar image at the current moment is input into the spatiotemporal convolution module for feature extraction to obtain the spatiotemporal feature map at the current moment; The predicted optical flow field between the current time and the previous time and the spatiotemporal feature map of the current time are input into the prediction module for prediction to obtain the predicted radar map of the next time. The predicted radar image for the next moment is used to predict the cloud distribution and precipitation conditions for the next moment; the radar image extrapolation prediction model is determined based on the optical flow extraction network and the spatiotemporal convolutional network.

8. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the radar image extrapolation prediction method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the radar image extrapolation prediction method as described in any one of claims 1 to 6.

Images