Radar echo meteorological data simulation prediction generation method and device, electronic equipment and storage medium

By constructing a video generation method based on diffusion model and RoBERTa model, the problem of poor prediction effect in short-term rainfall prediction is solved, and more accurate and stable radar echo video generation is achieved, which improves the accuracy and efficiency of short-term rainfall prediction.

CN119007070APending Publication Date: 2024-11-22INST OF PHYSICS HENAN ACAD OF SCI +1
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411051998.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The prior art has poor prediction effect and poor generalization in short-term rainfall prediction, making it difficult to accurately capture the occurrence and development of strong convection at short-term and small scales. Traditional methods lack physical foundation and have low forecast accuracy.

Method used

Build a video generation model based on the improved diffusion model, combine the RoBERTa model to predict semantic information, generate inter-frame change sequences of radar echo video, and fuse it with the initial frame, and extract content features and Inter-frame change information is predicted and trained through the Gaussian diffusion model.

Benefits of technology

It improves the accuracy and efficiency of radar echo video generation, enhances the model's perception of semantic features, realizes more accurate and stable analysis of short-term rainfall prediction, and improves the effect and interpretability of video inference generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007070A_ABST
    Figure CN119007070A_ABST
Patent Text Reader

Abstract

The invention discloses a radar echo meteorological data simulation prediction generation method and device, electronic equipment and a storage medium. The method comprises the following steps: firstly, collecting a real radar echo video and a text label of the video as a training set; constructing a RoBERTa model to carry out feature extraction on the text annotation data; for the collected radar echo video data, utilizing a background difference method and an inter-frame difference method to extract inter-frame change information; constructing an image encoder based on Transform to extract spatial features of the initial frame; building a diffusion model, fusing the initial frame features and the text features, inputting the fused features as condition information into the diffusion model, and inputting the inter-frame change sequence into the diffusion model for forward diffusion; performing reverse denoising on the diffusion model, predicting added noise calculation loss, and training the diffusion model; and inputting a test sample into the trained diffusion model for denoising and sampling to obtain a predicted inter-frame change sequence, combining the predicted inter-frame change sequence with the initial frame of the test sample, and predicting and generating a future radar echo video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer information technology, and in particular relates to a radar echo meteorological data simulation prediction generation method, device, electronic equipment and storage medium. Background Art

[0002] Accurate and efficient weather forecasts are of great significance for supporting agricultural production, ensuring public safety, promoting traffic safety, supporting energy management, serving national defense and military, and improving the quality of life of the public. Short-term rainfall forecasts have always been a key issue of concern and research in major natural disasters. By continuously improving the accuracy and timeliness of weather forecasts, we can better respond to natural disasters, optimize resource utilization, protect life and property, and promote the sustainable development of the economy and society.

[0003] However, due to the wide range of meteorological data sources, large data volume, high dimensionality, and nonlinear characteristics and high uncertainty of the atmospheric system, short-term rainfall prediction often faces great challenges. Radar echo data has strict time series characteristics and can more intuitively and efficiently reflect the actual precipitation situation. The prediction and analysis of radar echo data can effectively improve the ability to monitor disaster weather and improve the quality of weather forecasts. However, in actual applications, only experienced forecasters can give relatively accurate qualitative forecasts, which are highly subjective.

[0004] Traditional short-term rainfall prediction mainly includes statistical forecast, numerical model forecast and radar extrapolation method. These methods lack physical basis, have low forecast accuracy, and cannot accurately capture the occurrence and development of short-term, small and medium-scale severe convection. At present, with the rapid development of artificial intelligence technology, deep learning models can be trained to predict and analyze future development based on historical spatiotemporal information, and have been widely used in many fields. The rapid iterative development of generative artificial intelligence technology is causing a boom in the field of computer vision. Shi En et al. proposed a method based on dynamic convolutional neural network to realize the extrapolation of radar echoes, and its performance is better than the traditional methods COTREC and DITERC. Shi Xingjian et al. proposed a convolutional long short-term memory network model, which can simultaneously learn the spatial and temporal features of radar images, solve the echo extrapolation problem, and obtain more accurate results than the real-time optical flow method. On this basis, Zhang Lingling connected the convolutional autoencoder with ConvLSTM to enhance the quality of the extrapolated image. The current method has improved the prediction effect to a certain extent, but there is still a problem of low accuracy.

[0005] The patent number is CN2023116590135, and the invention name is a Chinese patent for a multi-frequency analysis video generation method based on attention mechanism. A generation model based on attention mechanism and wavelet analysis is constructed to generate autonomous driving scenes. Since it uses a cascaded temporal wavelet analysis module, the model is more suitable for longer scene generation and poses great challenges in short-term rainfall prediction. Summary of the invention

[0006] The present invention proposes a radar echo meteorological data simulation prediction generation method, device, electronic device and storage medium to solve the problems of poor prediction effect and weak generalization in the prior art. By constructing a video generation model based on an improved diffusion model, and predicting and generating an inter-frame change sequence of radar echo video according to semantic information, and then fusing the inter-frame change sequence with the initial frame to obtain a generated short-time radar echo video sequence, the Bert model is introduced in the process to encode text information, so that the model can better pay attention to semantic features and improve training efficiency.

[0007] The technical solution of the present invention is as follows:

[0008] A radar echo meteorological data simulation prediction generation method comprises the following steps:

[0009] S1, build a radar echo meteorological video database and obtain video sequences from the database;

[0010] S2, preprocessing the radar echo video data, and dividing the video sequence data set into a training set and a test set;

[0011] S3, build Unet-3D network model;

[0012] S4. Build the RoBERTa model and use the pre-trained model weights to encode the text information as the conditional information X to control video generation. text ;

[0013] S5. Build a Transformer-based image encoder model and input the initial frame of the video into the image encoder to extract content features X from the initial frame. fea ;

[0014] S6, construct a Gaussian diffusion model, and integrate the RoBERTa model in S4 and the encoder model in S5 into the model to predict the inter-frame change sequence;

[0015] S7, construct the overall model M;

[0016] S8. Use the training dataset prepared in S1 to train the overall model M in S7:

[0017] S9, the generated inter-frame change sequence Fusion with the initial frame information to obtain the generated video sequence

[0018] S10, the real video sequence S = {x1, x2, ..., x t} and the generated video frame sequence Compare and calculate the loss, and add the noise loss in S8 to the reconstruction loss of the generated video sequence in a certain ratio to obtain the comprehensive loss, and then use the back propagation algorithm to train the overall model M;

[0019] S11. After the model training is completed, the initial image x0, the text control information C and the time step t are input to the overall model M, thereby obtaining the inter-frame change sequence generated by the Gaussian diffusion model Adding it to the initial input image x0, we can get the predicted radar echo video sequence

[0020] Furthermore, step S1 is specifically as follows:

[0021] The video sequence datasets are obtained from the radar echo video database. The datasets include the Sevir dataset and the HKO-7 dataset. When training the network, one of the datasets is used as the only dataset to extract a certain number of video frame sequences as input, and the subsequent video frames are used as the corresponding reference results. Then, the same operation is performed with the other dataset as the only dataset.

[0022] Furthermore, step S2 is specifically as follows:

[0023] S21. Cropping: The original training sample is randomly cut into a 640*640 pixel video sequence, the formula is: x = f reduce (x)+f crop (x), where f reduce Indicates deleting pixel information outside the specified area; f crop It means to find the minimum enclosing rectangle of the area with gray value information in the image, and then crop the image to the size of this rectangle.

[0024] S22, HSL adjustment: Multiply the hue, saturation, and brightness of the cropped sample by a random value δ∈[1.0,1.2] to simulate the lighting changes in the natural environment.

[0025] S23. Then, the video sequence dataset is divided into a training set and a test set.

[0026] Furthermore, step S3 is specifically as follows:

[0027] Using the Unet-2D network structure, the 2D convolution is replaced by 3D convolution, and the original 3x3 convolution is replaced by 1x3x3 convolution, where the first dimension corresponds to the video frame, i.e. the time dimension, and the second and third dimensions correspond to the frame height and frame width, i.e. the spatial dimension, making it suitable for video data and constructing the denoising module of the diffusion model; the model mainly involves the 3D convolution layer Conv3D (C in ,C out ,k,s,p), activation layer ReLU, pooling layer Maxpooling3D(2,2), upsampling layer UpSample3D(2), where C in is the number of input channels, C out is the number of output channels, k is the convolution kernel size, s is the stride, and p is the padding. In addition, there are some jump link layers

[0028] Further, step S4 is specifically as follows:

[0029] Construct multiple Transformer encoder blocks, each of which contains multiple self-attention layers and feed-forward neural network layers. Input a word sequence, process it through multiple layers of Transformer encoders, and output a word representation for each position.

[0030] The calculation formula of self-attention is: Where Q represents query, K represents key, V represents value, and d k Indicates the dimension of the vector used to scale the dot product.

[0031] The self-attention is divided into multiple heads, each of which focuses on learning different aspects of features. The divided multi-head attention can be expressed as:

[0032] MultiHead(Q,K,V)=Concat(head1,…,head h )W o , where h represents the number of attention heads, W o is the trainable output weight matrix.

[0033] Further, step S7 is specifically as follows:

[0034] In this model, Unet3D in S3 and Gaussian diffusion model in S6 are instantiated, input-output and loss functions are established, and the predicted inter-frame change sequence is combined with the initial input image to obtain the generated video sequence.

[0035] Further, step S8 is specifically as follows:

[0036] Extract t-frame continuous video image sequence X = {x1, x2, ..., x t}, use the background difference method to obtain background modeling to obtain background information Z, use the frame difference method to calculate the frame difference sequence R of adjacent frames, and fuse the background modeling result and the frame difference information to obtain the frame change sequence D = {d1, d2, ..., d t-1}, the fusion method used is:

[0037]

[0038] The video image sequence X, the frame change sequence D, the text control information C, and the time step t are input into the overall model M constructed in S7 to generate the frame change sequence At the same time, the content feature map X0, the inter-frame change sequence D, and the text control information C generated in S5 are input into the Gaussian diffusion model to obtain the loss of the prediction noise and generate the predicted inter-frame change sequence

[0039] Furthermore, step S10 is specifically as follows:

[0040] The mixed loss functions involved in training are:

[0041] Generate graph sequence reconstruction loss function:

[0042]

[0043] Prediction noise loss function:

[0044] Structural similarity loss function: where μ x and They are x and The average value of and They are x and The covariance of , C1 and C2 are constants used for stability.

[0045] Perform a weighted mix of the above loss functions to obtain a mixed loss:

[0046] L mixed =αL pixel +βL pixel +γL ssim , α, β and γ are the weights of each function respectively. These hyperparameters can be adjusted during the experiment to achieve better experimental results.

[0047] A radar echo meteorological data simulation prediction generation device comprises an acquisition module and a prediction module, wherein the acquisition module is used to acquire a video sequence data set from a database, and the prediction module is used to generate the prediction method.

[0048] An electronic device includes a memory and a processor, wherein the memory and the processor are connected via a communication bus, and the communication bus is connected to an external device via a communication interface. The memory stores a computer program, and the processor is configured to run the computer program to execute the above method.

[0049] The processor may be a microprocessor, and the memory may include a memory, such as a random access memory (RAM), or a non-volatile memory, etc. Of course, the electronic device may also be provided with other hardware as required.

[0050] A storage medium stores a plurality of instructions, which can be loaded by a processor to execute the above method.

[0051] Compared with the prior art, this application has the following beneficial effects:

[0052] 1. The present invention proposes a radar echo meteorological data simulation prediction generation method, device, electronic device and storage medium. The existing statistical forecasting methods lack physical basis and have low forecast accuracy. Although the numerical model method can predict the rainfall range and trend in the medium and long term, its time and space resolution is relatively low compared with the small and medium scale severe convective weather system, and it is difficult to accurately capture the occurrence and development of severe convection in a short period of time. The relatively mature extrapolation methods such as cross-correlation method and optical flow method do not have historical learning capabilities, and parameter setting is relatively difficult. The radar echo meteorological data simulation prediction generation model we proposed can directly generate inter-frame change information between adjacent frames based on semantic information and observation data, and then directly fuse the inter-frame change information with the source image to obtain the generated radar echo video. The model makes full use of the temporal and spatial characteristics of radar echo data and improves the accuracy and efficiency of generation.

[0053] 2. The present invention introduces the RoBERTa model as a semantic information processing model to further enhance the model's ability to perceive feature information contained in text information. As a mature and widely used pre-trained model, the RoBERTa model is used to process language and can better focus on the features hidden in each word and the information implied in the sentence itself, and extract the features as semantic control information predicted by the model. The application of the RoBERTa model makes the generated video frames more consistent with the meaning expressed by the language, further improving the accuracy and effect of video reasoning generation.

[0054] 3. The present invention introduces the RoBERTa model and diffusion mechanism to predict the inter-frame change sequence, thereby generating a video sequence, which not only realizes a comprehensive analysis and accurate extraction of semantic information, but also ensures that video frames are generated quickly and conveniently. This method not only improves the accuracy and stability of video generation, but also enables a better understanding of the implicit feature information in the language and improves the interpretability of the model. The present invention has important application value and broad development prospects in the fields of radar weather forecasting and short-term rainfall analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of the radar echo meteorological data simulation prediction generation process of the present invention;

[0056] Figure 2 It is a diagram of the implementation process of the present invention;

[0057] Figure 3 This is a schematic diagram of the training structure of the radar echo meteorological data simulation prediction generation model of the present invention;

[0058] Figure 4 This is a schematic diagram of the sampling structure of the radar echo meteorological data simulation prediction generation model described in the present invention.

[0059] Figure 5 It is a structural schematic diagram of a radar echo meteorological data simulation prediction generating device provided by an example of the present invention;

[0060] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an example of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] A radar echo meteorological data simulation prediction generation method comprises the following steps:

[0063] S1, build a radar echo meteorological video database and obtain video sequences from the database;

[0064] Step S1 is specifically as follows:

[0065] The video sequence datasets are obtained from the radar echo video database. The datasets include the Sevir dataset and the HKO-7 dataset. When training the network, one of the datasets is used as the only dataset to extract a certain number of video frame sequences as input, and the subsequent video frames are used as the corresponding reference results. Then, the same operation is performed with the other dataset as the only dataset.

[0066] S2, preprocessing the radar echo video data, and dividing the video sequence data set into a training set and a test set; step S2 is specifically:

[0067] S21. Cropping: The original training sample is randomly cut into a 640*640 pixel video sequence, the formula is: x = f reduce (x)+f crop (x), where f reduce Indicates deleting pixel information outside the specified area; f crop It means to find the minimum enclosing rectangle of the area with gray value information in the image, and then crop the image to the size of this rectangle.

[0068] S22, HSL adjustment: Multiply the hue, saturation, and brightness of the cropped sample by a random value δ∈[1.0,1.2] to simulate the lighting changes in the natural environment.

[0069] S23. Then, the video sequence dataset is divided into a training set and a test set.

[0070] S3, build Unet-3D network model,

[0071] Using the Unet-2D network structure, the 2D convolution is replaced by 3D convolution, and the original 3x3 convolution is replaced by 1x3x3 convolution, where the first dimension corresponds to the video frame, i.e. the time dimension, and the second and third dimensions correspond to the frame height and frame width, i.e. the spatial dimension, making it suitable for video data and constructing the denoising module of the diffusion model; the model mainly involves the 3D convolution layer Conv3D (C in ,C out ,k,s,p), activation layer ReLU, pooling layer Maxpooling3D(2,2), upsampling layer UpSample3D(2), where C in is the number of input channels, C out is the number of output channels, k is the convolution kernel size, s is the stride, and p is the padding. In addition, there are some jump link layers

[0072] S4. Build the RoBERTa model and use the pre-trained model weights to encode the text information as the conditional information X to control video generation.text ;

[0073] Specifically, multiple Transformer encoders are constructed, each of which contains multiple self-attention layers and feed-forward neural network layers. The word sequence is input and processed by the multi-layer Transformer encoder, and the output contains the word representation of each position.

[0074] The calculation formula of self-attention is: Where Q represents query, K represents key, V represents value, and d k Indicates the dimension of the vector used to scale the dot product.

[0075] The self-attention is divided into multiple heads, each of which focuses on learning different aspects of features. The divided multi-head attention can be expressed as:

[0076] MultiHead(Q,K,V)=Concat(head1,…,head h )W o , where h represents the number of attention heads, W o is the trainable output weight matrix.

[0077] S5. Build a Transformer-based image encoder model and input the initial frame of the video into the image encoder to extract content features X from the initial frame. fea .

[0078] S6, construct a Gaussian diffusion model, and integrate the RoBERTa model in S4 and the encoder model in S5 into the model to predict the inter-frame change sequence;

[0079] S7, construct the overall model M,

[0080] In this model, Unet3D in S3 and Gaussian diffusion model in S6 are instantiated, input-output and loss functions are established, and the predicted inter-frame difference sequence is combined with the initial input picture to obtain the generated video sequence.

[0081] S8. Use the training dataset prepared in S1 to train the overall model M in S7:

[0082] Extract t-frame continuous video image sequence X = {x1, x2, ..., x t}, use the background difference method to obtain background modeling to obtain background information Z, use the frame difference method to calculate the frame difference sequence R of adjacent frames, and fuse the background modeling result and the frame difference information to obtain the frame change sequence D = {d1, d2, ..., d t-1}, the fusion method used is:

[0083]

[0084] The video image sequence X, the frame change sequence D, the text control information C, and the time step t are input into the overall model M constructed in S7 to generate the frame change sequence At the same time, the content feature map X0, the inter-frame change sequence D, and the text control information C generated in S5 are input into the Gaussian diffusion model to obtain the loss of the prediction noise and generate the predicted inter-frame change sequence

[0085] S9, the generated inter-frame change sequence Fusion with the initial frame information to obtain the generated video sequence

[0086] S10, the real video sequence S = {x1, x2, ..., x t} and the generated video frame sequence Compare and calculate the loss, and add the noise loss in S8 to the reconstruction loss of the generated video sequence in a certain ratio to obtain the comprehensive loss, and then use the back propagation algorithm to train the overall model M;

[0087] The mixed loss functions involved in training are:

[0088] Generate graph sequence reconstruction loss function:

[0089]

[0090] Prediction noise loss function:

[0091] Structural similarity loss function: where μ x and They are x and The average value of and They are x and The covariance of , C1 and C2 are constants used for stability.

[0092] Perform a weighted mix of the above loss functions to obtain a mixed loss:

[0093] L mixed =αL pixel +βL pixel +γL ssim , α, β and γ are the weights of each function respectively. These hyperparameters can be adjusted during the experiment to achieve better experimental results.

[0094] S11. After the model training is completed, the initial image x0, the text control information C and the time step t are input to the overall model M, thereby obtaining the inter-frame change sequence generated by the Gaussian diffusion model Adding it to the initial input image x0, we can get the predicted radar echo video sequence

[0095] The embodiment of the present application also provides a radar echo meteorological data simulation prediction generation device, such as Figure 5 As shown, it includes an acquisition module 10 and a prediction module 20, wherein the acquisition module 10 is used to acquire a video sequence data set from a database, and the prediction module 20 is used to generate a predicted future video frame.

[0096] The embodiment of the present application also provides an electronic device, see Figure 6 The electronic device includes a memory 730 and a processor 710, the memory 730 and the processor 710 are connected via a communication bus 740, and the communication bus 740 is connected to an external device via a communication interface 720. The memory 730 stores a computer program, and the processor 710 is configured to run the computer program to execute the above method.

[0097] The processor 710 may be a microprocessor, and the memory 730 may include a memory, such as a random access memory (RAM), or a non-volatile memory, etc. Of course, the electronic device may also be provided with other hardware as required.

[0098] An embodiment of the present application also provides a storage medium, in which multiple instructions are stored. The instructions can be loaded by a processor to execute a radar echo meteorological data simulation prediction generation method provided by an embodiment of the present application.

[0099] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A radar echo meteorological data simulation prediction generation method, characterized in that: The following steps are involved: S1, build a radar echo meteorological video database and obtain video sequences from the database; S2, preprocessing the radar echo video data, and dividing the video sequence data set into a training set and a test set; S3, build Unet-3D network model; S4. Build the RoBERTa model and use the pre-trained model weights to encode the text information as the conditional information X to control video generation. text ; S5. Build a transformer-based image encoder model and input the initial frame of the video into the image encoder to extract content features X from the initial frame. fea ; S6, constructing a Gaussian diffusion model, and integrating the Bert model in S4 and the encoder model in S5 into the model to predict the inter-frame change sequence; S7, construct the overall model M; S8. Use the training dataset prepared in S1 to train the overall model M in S7: S9, the generated inter-frame change sequence Fusion with the initial frame information to obtain the generated video sequence S10, the real video sequence S = {x1, x2, ..., x t } and the generated video frame sequence Compare and calculate the loss, and add the noise loss in S8 to the reconstruction loss of the generated video sequence in a certain ratio to obtain the comprehensive loss, and then use the back propagation algorithm to train the overall model M; S11. After the model training is completed, the initial image x0, the text control information C and the time step t are input to the overall model M, thereby obtaining the inter-frame change sequence generated by the Gaussian diffusion model Adding it to the initial input image x0, we can get the predicted radar echo video sequence 2. A radar echo meteorological data simulation prediction generation device, characterized in that: It comprises an acquisition module (10) and a prediction module (20), wherein the acquisition module (10) is used to acquire a video sequence data set from a database, and the prediction module (20) is used to generate the radar echo meteorological data simulation prediction generation method described in claim 1.

3. An electronic device, characterized in that: The method comprises a memory (730) and a processor (710), wherein the memory (730) and the processor (710) are connected via a communication bus (740), and the communication bus (740) is connected to an external device via a communication interface (720). A computer program is stored in the memory (730), and the processor (710) is configured to run the computer program to execute the method according to claim 1.

4. A storage medium storing a plurality of instructions, wherein the instructions can be loaded by a processor to execute the radar echo meteorological data simulation prediction generation method according to claim 1.

Citation Information

Cited By

  • Diffusion model-based radar echo prediction method and apparatus, and electronic device

    CN119916326A

  • Radar echo extrapolation processing method and system based on hypergraph enhanced diffusion model

    CN120446903A

  • Radar echo extrapolation processing method and system based on hypergraph enhanced diffusion model

    CN120446903B