A method for predicting short-term cloud movement trends

Abstract

The present invention discloses a method for predicting short-term motion trend of clouds, comprising: obtaining a satellite cloud image sequence data set, and constructing a cloud short-term motion trend prediction model based on deep voxel flow and Vision Transformer; training the cloud short-term motion trend prediction model based on the satellite cloud image sequence data set to obtain a target prediction model; predicting the short-term motion trend of clouds through the target prediction model to obtain a prediction result. The present invention improves the accuracy and credibility of the prediction, can capture the details of cloud movement at different scales, and greatly enhances the perception of large-scale dynamic changes.

Description

Technical Field

[0001] The invention belongs to the field of cloud movement trend prediction, and in particular relates to a method for predicting short-term cloud movement trend. Background Art

[0002] The study of cloud movement is of vital importance to the fields of meteorology and climate science. The study of short-term cloud movement trends is of great value to the accuracy of weather forecasts and climate models. It can provide instant information about the development, movement and changes of clouds, which helps to more accurately predict weather conditions, such as the formation and path of precipitation, storms and typhoons; therefore, the study of short-term cloud movement trends is very necessary.

[0003] However, most of the research work on cloud motion prediction is focused on specific types of clouds or regions. In severe weather forecasting, more attention is often paid to the motion of convective clouds, while in short-term precipitation forecasting tasks, precipitation clouds are more important. At present, there is a lack of short-term motion trend prediction for all clouds in satellite cloud images. In addition, most of the current cloud motion trend prediction models are only applicable to a single type of image data, which limits their applicability and the use of complementary information between different data. This limitation not only increases the complexity and cost of research and application, but also limits the improvement of prediction accuracy. Therefore, the development of a comprehensive prediction model that can process and integrate multiple image types has become the key to improving the efficiency and accuracy of cloud motion prediction. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a method for predicting short-term cloud movement trends, comprising:

[0005] Obtain satellite cloud image sequence data sets and build a cloud short-term motion trend prediction model based on deep voxel flow and Vision Transformer;

[0006] Based on the satellite cloud image sequence data set, a cloud cluster short-term motion trend prediction model is trained to obtain a target prediction model;

[0007] The target prediction model is used to predict the short-term movement trend of the cloud cluster to obtain a prediction result.

[0008] Preferably, the cloud cluster short-term motion trend prediction model further includes five TGDVF sub-networks connected in series, and the i-th cloud cluster short-term motion trend prediction model is expressed as By combining the two frames I t-1 and I t Synthesized frames and the estimated voxel flow As input, learn to predict the target voxel flow At the same time, the last frame synthesized by the TGDVF sub-network will be used as input to the discriminator.

[0009] Preferably, the process of predicting the short-term movement trend of clouds by using the target prediction model includes:

[0010] For the cloud to be predicted, the predicted image is indirectly generated by optical flow mapping;

[0011] A scaling factor sequence is added to the target prediction model, and two channels are designed to extract the overall spatial features of the cloud image and the motion features of the cloud clusters using CNN and ViT-M respectively through a pyramid optical flow estimation method.

[0012] Using each layer of the target prediction model represented by the Laplacian pyramid, outputting the sum of the reconstruction losses as the loss function, and the PatchGAN discriminator to form a GAN network model, discriminating the local area in the predicted image based on the GAN network model to obtain the local change;

[0013] A prediction result of the short-term movement trend of the cloud to be predicted is obtained according to the local changes.

[0014] Preferably, the process of adding a scaling factor sequence to the target prediction model comprises:

[0015] Design scaling factor s according to the motion scale differences in different regions i , by scaling factor s i Perform downsampling;

[0016] The scaling factor s i Reduce image resolution and expand receptive field by downsampling;

[0017] Wherein, the scaling factor s i For a decreasing sequence:

[0018] I i =Downsample(I original , S i )

[0019] F i =OpticalFlowEstimation(I i )

[0020] F i，refined =Refine(F i-1 , S i ) Among them, I i is the downsampled image at the current level, s i is the scaling factor for the current level, F iis the estimated optical flow field at the current level, F i,refined It is the optical flow field refined by the optical flow field of the previous level.

[0021] Preferably, the process of performing optical flow estimation by the pyramid optical flow estimation method includes:

[0022] Starting from applying the largest scaling factor, downsample the original image to obtain a small image version;

[0023] Performing optical flow estimation based on the small image version to obtain a preliminary estimate of the optical flow field;

[0024] Then the scaling factor is gradually reduced, the resolution of the image is gradually increased, and the optical flow field is refined and adjusted accordingly. The previously estimated optical flow field at each step is used as the initial condition for the estimation of the next resolution level to obtain the final estimate of the optical flow field.

[0025] Preferably, the process of designing two channels to use CNN and ViT-M to extract the overall spatial features of the cloud image and the cloud group motion features includes:

[0026] A dual-branch network structure is constructed by using a pyramid optical flow estimation method. One channel is performed at high resolution to extract features from the image at the original resolution to obtain spatial information. The other channel first uses bilinear interpolation to scale the image according to the scaling factor. Subsequently, ViT-M is used to extract the motion features of the local cloud clusters after segmentation. After that, the feature map is restored to the same dimension as the other channel through upsampling and then merged. Finally, the voxel flow is generated through deconvolution. It means that the output of the i-th TGDVF has the following relationship:

[0027]

[0028] in, and The initial value of is set to zero.

[0029] Preferably, the ViT-M modifies ViT by setting the moving step of the rectangular area to half of the rectangular frame, and then after selecting an area, sequentially translates the area by 8 units; for the features of the overlapping extracted areas, the weighted average method is used to integrate them to obtain ViT-M.

[0030] Preferably, the loss function is the reconstruction loss output for each block The sum of is expressed as:

[0031]

[0032] Preferably, the PatchGAN discriminator includes several convolutional layers, the activation function of each convolutional layer adopts the LeakyRelu activation function, the final convolution result is input into the fully connected layer, the output layer adopts the sigmoid activation function, and the discriminator uses It+1 generated by the generator as input to distinguish true from false, with 1 representing "true" and 0 representing "false". When the model converges, the discriminator judges the predicted frame as 0 and the real frame as 1.

[0033] Preferably, the PatchGAN discriminator is used for each N×N size patch in the image, and the PatchGAN discriminator attempts to determine whether the patch is real or fake;

[0034]

[0035] Where I is the input image, I m is the mth patch in the image, M is the total number of patches in the image, and D patch It is the judgment function of the PatchGAN discriminator for a single patch.

[0036] Compared with the prior art, the present invention has the following advantages and technical effects:

[0037] The present invention adopts an advanced hybrid feature extraction strategy, uses a convolutional neural network (CNN) to extract global spatial information, and extracts cloud motion information based on ViT-M of Vision Transformer. The hybrid feature extraction strategy is used as the core part of the generator design, so that the generator can effectively capture the detailed changes of clouds and their motion trends. PatchGAN is selected as the discriminator to predict the motion trend of cloud images with higher quality, and it can distinguish the local areas of the image, which makes the model more sophisticated in processing image details, further improving the accuracy and credibility of the prediction. In solving the problem of large-scale motion prediction of clouds, by introducing a sequence of scaling factors, the model can capture the details of cloud motion at different scales, greatly enhancing the perception of large-scale dynamic changes. In the design of the loss function, the L1 loss calculated on each pair of images is represented by a Laplacian pyramid to collect information at each layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 This is a schematic diagram of the structure of a cloud short-term motion trend prediction model according to an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the receptive field structure of an embodiment of the present invention;

[0041] Figure 3 Schematic diagram of the discriminator structure of an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the module structure of a cloud short-term motion trend prediction model according to an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of dividing an original image into blocks according to an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of the prediction results of HiwDataset according to an embodiment of the present invention;

[0045] Figure 7 This is a comparison chart of the prediction results of the Shanghai radar echo data set according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] Embodiment 1

[0049] like Figure 1-7 As shown, this embodiment provides a method for predicting short-term cloud movement trends, including:

[0050] Obtain satellite cloud image sequence data sets and build a cloud short-term motion trend prediction model based on deep voxel flow and Vision Transformer;

[0051] Based on the satellite cloud image sequence data set, a cloud cluster short-term motion trend prediction model is trained to obtain a target prediction model;

[0052] The target prediction model is used to predict the short-term movement trend of the cloud cluster to obtain a prediction result.

[0053] In order to better study the cloud movement trends in various types of cloud images, this embodiment uses the Himawari-8 (Sunflower 8) meteorological geostationary satellite to create the HiwDataset satellite cloud image sequence data set, which includes a total of 4128 satellite cloud image sequences.

[0054] For the short-term motion trend prediction task of clouds, this embodiment innovatively proposes a short-term motion trend prediction model TGDVFN based on deep voxel flow and Vision Transformer. The model indirectly generates the predicted image by optical flow mapping, and the added scaling factor sequence ensures the versatility of the model for large-scale and small-scale tasks; then, with the help of the pyramid optical flow estimation method, two channels are designed to use CNN and ViT-M to extract the overall spatial features of the cloud image and the motion features of the cloud respectively; then, this embodiment uses the sum of the reconstruction losses of each layer of TGDVF output represented by the Laplacian pyramid as the loss function; finally, the model uses the PatchGAN discriminator to form a GAN network model structure to discriminate the local areas in the image, capture the local changes more finely, and improve the accuracy of the prediction. TGDVFN has strong competitiveness in the short-term motion trend prediction task of clouds, and the MS-SSIM reaches 98.47 and LPIPS reaches 3.05 in the prediction task of t+1 frame on the HiwDataset dataset.

[0055] This embodiment mainly analyzes and selects methods for the task of short-term cloud motion trend prediction. On this basis, a TGDVFN model is designed, and the specific design details in the model are described. Then, a comparative experiment is conducted between TGDVFN and other advanced models in HiwDataset and Shanghai radar echo dataset, and the experimental results are analyzed. Finally, a model generalization experiment and experimental results analysis are conducted in the KITTI video dataset.

[0056] The goal of cloud movement trend prediction is to predict the future cloud movement based on the currently available information. This task helps people study the motion characteristics of clouds, and combines the parameter information of the cloud anvil and base width, cloud top and cloud bottom height, cloud top buoyancy and other parameters in the cloud system to explore the complex relationship between the various parameters and the effect on the cloud. It is more commonly used for tracking and prediction of mesoscale convective systems and long-term and short-term precipitation forecasts. In the task of cloud movement trend prediction, short-term precipitation forecast is one of the most characteristic and practical subtasks. In this task, most research works regard short-term precipitation forecast as a radar sequence prediction task, and radar sequence prediction belongs to video prediction, or spatiotemporal sequence prediction, image sequence prediction subtask. Therefore, the models used for video prediction are generally applicable to radar sequence prediction tasks, and are also applicable to satellite cloud image sequence prediction tasks.

[0057] In this embodiment, the cloud motion trend prediction task is regarded as video prediction: given a sequence of past t frames Video prediction aims to predict future frames ~It+1, ~It+2, ~It+3, ... In order to reduce the input amount, the input of the video prediction model designed in this embodiment is only two consecutive frames I t-1 and I t It is expected that the prediction effect of ~It+1 will be the best, and then the future frames ~It+2, ~It+3, ... will be predicted through iteration. The video prediction model is represented as O θ (I t-1 ,I t ), where θ is the set of model parameters to be learned, and the learning goal is to convert ~It+1=O θ (I t-1 ,I t ) and “groundtruth” t+1 Minimize the difference between them.

[0058] Traditional optical flow methods can accurately estimate the motion between pixels in an image sequence, which is very important for tracking motion in videos and estimating speed in real-world scenes. By accurately calculating the movement between pixels, optical flow methods can maintain details and avoid blur when synthesizing or predicting video frames. Deep learning models are able to process and simulate highly complex data patterns, which enables them to simulate complex motion and appearance changes in video prediction. At the same time, neural networks can self-learn and extract features from large amounts of data, which means that they can learn how to predict new frames by observing a large number of video frames without the need for explicit rules. The Deep Voxel Flow (DVF) of this embodiment combines the accuracy and detail preservation capabilities of traditional optical flow methods with the self-learning ability and complexity processing capabilities of neural networks. Voxel flow includes predictions of the results of object motion and occlusion, creating a method that can produce high-quality video frames.

[0059] The DVF model structure of this embodiment takes into account the local consistency of time and space, and the pixels of the generated future frame generally come from the nearby area of ​​the previous frame. t+1 To I t Estimating optical flow Then, the pixel-wise inverse deformation (denoted as ) to get the corresponding frame. In addition, in order to deal with occlusion, some methods further introduce fusion graph m to fuse I t and I t-1 The final predicted frame is given by the following consensus:

[0060]

[0061] TGDVFN, a cloud short-term motion trend prediction model based on deep voxel flow;

[0062] For the video prediction model, clouds will deform, separate, generate and dissipate due to various reasons during the movement. At this time, the local feature extraction ability of the CNN network model is insufficient for such complex movements. Convolution is designed to extract spatial features in images and capture local spatial dependencies. It has a natural disadvantage in capturing changes between frames. In such a difficult spatiotemporal sequence prediction task, using only convolution cannot cope with complex nonlinear spatiotemporal transformations. The advantage of the Transformer model in capturing global dependencies can make up for this shortcoming and more accurately predict complex cloud movements. The GAN network model structure allows the generator and the discriminator to compete with each other during the training process to continuously improve their respective performance. The addition of the discriminator as an effective constraint on the generation of model prediction frames will have a positive impact on the effect of the model. Therefore, the model design idea of ​​this embodiment is to introduce a hybrid network model of Transformer and GAN on the basis of the former's voxel flow estimation, increase the model's global information capture ability, and design a network model that is more suitable for cloud motion prediction. The voxel flow generation adversarial network proposed in this embodiment is as follows: Figure 1 As shown in Figure 1, TGDVFN contains five TGDVF sub-networks connected in series to form a generator and a discriminator module composed of PatchGAN.

[0063] Furthermore, the generator module consists of five TGDVF sub-networks connected in series, and the input of the generator module is only two consecutive satellite cloud images in the sequence. t-1 and I t , the image ~It+1 output by the last stage TGDVF is the predicted image of the prediction model proposed in this embodiment.

[0064] When designing the model, it was observed that the visualization images of the Sunflower-8 data satellite cover a very wide geographical range, from -60.0 degrees to 60.0 degrees in latitude and from 80.0 degrees to 200.0 degrees in longitude. This means that cloud image motion observations in different regions and at different time periods will show different motion scales. This kind of change is common in large-scale meteorological monitoring and analysis, especially when dealing with such a wide geographical range. In order to effectively process and analyze this data, considering the possible differences in motion scales in different regions, this embodiment designs a scaling factor s i , by scaling factor s i Downsampling is performed. The scaling factor adjusts the image through downsampling technology. The purpose of this is to expand the receptive field and effectively capture the details of movement and changes even in a large range of observations. The receptive field diagram is shown in Figure 2 As shown. Downsampling is a processing method to reduce the amount of data by reducing the image resolution, which helps the model or analysis method focus on larger-scale patterns and features. This method not only helps to improve the efficiency of data processing, but also reduces the demand for computing resources while maintaining data quality. It is an effective strategy for processing and analyzing large-scale meteorological data.

[0065] When dealing with the problem of optical flow estimation, a common strategy is to first predict the optical flow field on a small-sized image, and then gradually refine the prediction results to a larger-sized image. This approach allows researchers to make a preliminary estimate at a low computational cost, and then improve the accuracy and details of the optical flow field by gradually increasing the image resolution. In order to implement this process more efficiently and simplify the problem for intuitive understanding, this embodiment also uses a sequence of decreasing scaling factors. In the end, the scaling factors are actually set to [4, 4, 2, 2, 1].

[0066] I i = Downsample(Ioriginal, S i ) (4)

[0067] F i =OpticalFlowEstimation(I i ) (5)

[0068] F i，refined =Refine(F i-1 , S i ) (6)

[0069] Among them, I i is the downsampled image at the current level, s i is the scaling factor for the current level, F i is the estimated optical flow field at the current level, F i,refined It is the optical flow field refined by the optical flow field of the previous level.

[0070] Specifically, the process starts by applying the largest scaling factor, which significantly downsamples the original image to produce a smaller version of the image. Optical flow estimation is performed on this smaller image, and a preliminary estimate of the optical flow field can be obtained more quickly due to the reduction in image size. Subsequently, the resolution of the image is gradually increased by gradually reducing the scaling factor, while the optical flow field is correspondingly refined and adjusted to adapt to the changes in details at higher resolutions. At each step, the previously estimated optical flow field is used as the initial condition for the estimation at the next resolution level, which ensures that the coherence and accuracy of the optical flow field gradually improve throughout the process.

[0071] Furthermore, the discriminator is composed of multiple convolutional layers. The activation function of each convolution layer uses the LeakyRelu activation function. The final convolution result is input into the fully connected layer. The output layer uses the sigmoid activation function. The discriminator converts the I generated by the generator into t+1 As input, the discriminator distinguishes true from false, with 1 representing "true" and 0 representing "false". When the model converges, the discriminator should distinguish the predicted frame as 0 and the real frame as 1. The structural diagram of the discriminator module is shown in Figure 3 shown.

[0072] In the process of selecting the discriminator, it is considered that the model proposed in this embodiment generates images by voxel flow mapping, which means that the generated images are "correct" in the macro sense, that is, they are consistent with the real images in terms of overall layout and composition. Therefore, the design of the model does not prefer a discriminator that can better evaluate the authenticity of the entire image, but requires it to pay more attention to each small area in the image and judge whether the details of these small areas correspond to the real world. The design of the Markov discriminator (PatchGAN) cleverly meets this requirement because it focuses on the local features of the image rather than the whole. Unlike traditional discriminators, it divides the image into small patches (patches) and evaluates the authenticity of each patch. This method enables PatchGAN to capture the details and texture differences of the image more sensitively, thereby making a more refined identification of the quality of the generated image. PatchGAN can be described by the following simplified formula: For each N×N patch in the image, the PatchGAN discriminator D tries to determine whether the patch is real or fake. It can be expressed as:

[0073]

[0074] Where I is the input image, I m is the mth patch in the image, M is the total number of patches in the image, and D patch is the judgment function of the PatchGAN discriminator for a single patch. Choosing PatchGAN as the discriminator not only promotes the model to generate delicate and detailed images, but also ensures the local authenticity of the generated images while maintaining high efficiency. This in-depth attention to detail is an indispensable part of the short-term motion trend prediction task of clouds, and the application of PatchGAN just meets this point, making it a key choice for optimizing the detail expression of the TGDVFN model.

[0075] Furthermore, the TGDVFN proposed in this embodiment includes five TGDVF sub-networks connected in series, and the i-th TGDVF is represented as It combines the two frames I t-1 and It Synthesized frames and the voxel flow estimated by the previous block As input, we learn to predict the target voxel flow At the same time, the last frame synthesized by the TGDVF sub-network will be used as input to the discriminator.

[0076] The structure of TGDVF is as follows Figure 4 As shown. In order to capture large motions while retaining the original spatial information, a dual-branch network structure is constructed in the feature extraction part of this embodiment. This design is inherited from the pyramid optical flow estimation method. One of the channels is performed at high resolution, and features are extracted from the image at the original resolution to obtain spatial information. In the other channel, the bilinear interpolation method is first used to scale the image according to the scaling factor. Subsequently, the motion features of the local cloud clusters after segmentation are extracted through ViT-M. After that, the feature map is restored to the same dimension as the other channel through upsampling, and then the feature maps are merged.

[0077] Finally, the voxel flow is generated through deconvolution.

[0078] Use here represents the output of the i-th TGDVF, and has the following relationship:

[0079]

[0080] and The initial value of is set to zero.

[0081] In video prediction tasks, the movement scale of an object is affected by many factors, such as the speed of the object, the relative distance from the camera, etc. For an object with the same speed, if it is far away from the camera, its movement scale in the video will be relatively small; conversely, if the object is close to the camera, its movement scale will increase significantly, and sometimes it may even move out of the picture instantly. This phenomenon is very common in daily video capture, especially when using a fixed camera to shoot dynamic scenes.

[0082] However, when turning to the analysis of satellite cloud images, the situation is different. Satellite cloud images usually cover a wide geographical range, and the position of the synchronous satellite relative to the ground (or cloud layer) is relatively fixed. This means that under natural conditions and technical limitations, there is a natural threshold for the maximum movement scale of the cloud layer on the satellite cloud image. This threshold is determined by multiple factors, including the observation angle of the satellite, the actual movement speed and direction of the cloud layer, and the size of the observed area. When analyzing satellite cloud images, if you pay too much attention to dynamic changes in a large range, it will not only lead to a sharp increase in the amount of calculation, but also may increase the interference introduced by factors such as observation angle and weather conditions. Therefore, when designing video prediction models or analyzing satellite data, it is crucial to reasonably set the scale range of concern. This can not only improve the accuracy of the analysis, but also optimize the use of computing resources, thereby achieving efficient and reliable performance in large-scale data processing.

[0083] Therefore, in the specific design of the ViT-M module, this embodiment chooses to use Vision Transformer as the basic model for processing satellite cloud images. Compared with the traditional Transformer module, Vision Transformer can more effectively utilize the advantages of Transformer in processing image data, while avoiding some computing and storage challenges when processing large-scale image data. This embodiment sets the size of each patch to 16×16. When the image scaling factor is 4, an image will be divided into 8×8 rectangular areas, a total of 64 areas. Each rectangular area represents a size 16 times the original size of the image. The schematic diagram after block division is as follows Figure 5 As shown in Figure 1. The speed of cloud movement is affected by many factors, including atmospheric conditions, geographic location, and high and low pressure systems. Under certain conditions, such as typhoons, hurricanes, or other strong weather systems, the movement speed of clouds can be very fast. For general clouds, their movement speed generally follows the atmospheric wind speed, which can range from a few kilometers per hour to tens of kilometers per hour. However, in extreme weather conditions, such as typhoons or hurricanes, the movement speed of clouds can increase significantly. For example, the wind speed of a hurricane can exceed 250 kilometers per hour (about 155 miles per hour). Here, it is assumed that the maximum movement speed of clouds is 300 kilometers per hour to calculate the maximum distance that clouds can move on the surface of the earth. In the original image size, the actual physical length of a 16×16 rectangular area is about 1200 kilometers horizontally and about 400 kilometers vertically. The temporal resolution of the images in the dataset is 10 minutes. In a one-hour forecast task, the maximum movement distance of the cloud is less than 400 kilometers, so the size of this area meets the range discussion of the maximum natural threshold of cloud movement in the previous article.

[0084] In order to consider the problem of missing information at the boundary after region division, this embodiment modifies ViT by setting the moving step of the rectangular region area to half the side length of the rectangular frame, and then after selecting a region, the region is translated by 8 units in sequence. For the features of the overlapping extracted regions, the weighted average method is used for integration, which is called ViT-M in this embodiment.

[0085] Furthermore, in the TGDVF module, the training loss L of TGDVFN is total is the reconstruction loss of each block output The sum of:

[0086]

[0087] The Laplacian pyramid is a multi-scale image representation method that can capture and separate the details of an image at different scales. By considering the reconstruction losses at different scales (different levels represented by the Laplacian pyramid) and weighting these losses, this method enables the generator to not only focus on the reconstruction of high-resolution details during the learning process, but also focus on the reconstruction of image features at more abstract, lower resolution levels. Specifically, this embodiment uses the weighting factor y n-i to achieve.

[0088] The loss function G of the generator Loss Calculated by the following formula:

[0089]

[0090] Among them, L total is the result in formula (9), represents the negative logarithm of the probability that the image generated by the generator is considered by the discriminator to be a real image. Here λ is a weighting coefficient used to balance the two loss parts, and it is set to 0.5 during initial training. This term encourages the generator to generate images that are not only similar to real images at the pixel level, but also sufficiently similar to the real images as a whole to fool the discriminator into thinking that these generated images are real. Through this design, the generator is not only required to generate high-quality images at the pixel level (through L total ), and also generate images that are realistic enough at the feature level to fool the discriminator (via item).

[0091] This example also evaluates the effectiveness of the TGDVFN model in predicting tasks when the data set is large-scale satellite cloud images, radar echo images, and videos, and analyzes the experimental results. At the same time, TGDVFN is compared with more advanced methods in recent years.

[0092] The deep learning framework used in the experiment is pytorch. Pytorch can be seen as a numpy data package with GPU support. It can also be seen as a powerful deep neural network with automatic differentiation capabilities. It can not only achieve powerful GPU acceleration, but also support dynamic neural networks. In addition to its powerful functions, its way of defining network structure is relatively simple, intuitive and flexible, and the code is highly readable. The specific experimental environment configuration is shown in Table 1

[0093] Table 1

[0094]

[0095] First, we conduct experiments in our self-built dataset, HiwDataset. We randomly select three groups from the test set to display the prediction results. Figure 6 As shown. In the figure, seven consecutive satellite cloud images are used as a sequence, of which the upper half shows the actual observations, and the lower half shows the prediction results made by the TGDVFN model proposed in this embodiment when the time is t=1 and t=2 as input. By visual observation, it can be clearly seen that there is a high degree of similarity between the predicted image and the actual observations. It is worth noting that although the TGDVFN model only uses two frames of images as input, it can still predict future frames with a fairly high accuracy. The generated prediction results are consistent with the actual movement trend of the clouds, which further verifies the effectiveness and accuracy of the TGDVFN model in processing satellite cloud image prediction tasks, demonstrates the ability of the TGDVFN model in understanding and capturing the dynamic changes of clouds, and emphasizes its potential to accurately predict complex natural phenomena when relying only on limited input data.

[0096] In order to make a more rigorous and detailed comparison, this embodiment uses four parameters, namely, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Multi-Scale Structural Similarity (MS-SSIM), and Learned Perceptual Image Patch Similarity (LPIPS), as evaluation criteria to quantitatively compare the difference between the predicted image generated by the model proposed in this embodiment and the original image. In this way, the performance of the model and the quality of its generated images can be more comprehensively evaluated and understood from different dimensions and levels.

[0097] According to the average results of the four evaluation parameters obtained by TGDVFN after comprehensive analysis of the entire test set, it can be seen that in the cloud movement trend prediction task, the TGDVFN model has shown excellent performance in all indicators. Specifically, in the prediction of the last five frames, the peak signal-to-noise ratio (PSNR) of the predicted frames t+1, t+3, and t+5 of TGDVFN are 28.75, 23.37, and 20.78, respectively, the structural similarity (SSIM) is 91.19, 76.26, and 64.09, respectively, the multi-scale structural similarity (MS-SSIM) index is 98.47, 93.44, and 85.64, respectively, and the learned perceptual image quality evaluation standard (LPIPS) is 3.05 and 10.02, respectively. The four indicators prove that the predicted results and the real images are very similar in structure, brightness, and contrast, and this similarity is maintained at multiple scales, including structural information such as edges and textures. Satellite cloud images use grayscale values ​​to represent different cloud clusters. The high index of MS-SSIM also shows that the trend of the change of light and dark contrast in the image is similar at different scales, that is, the prediction accuracy of cloud clusters is high. The experimental results show that the TGDVFN model in this study can generate prediction images that are very close to the actual satellite cloud images in short-term prediction tasks, and the prediction effect is good.

[0098] This embodiment also uses HiwDataset to compare with other advanced models. For the sake of fairness, the input of all multi-input models is three continuous sequence images. In Table 2, TGDVFN shows better performance than other advanced models. Specifically, in the prediction of t+1 and t+3 frames, TGDVFN has the best MS-SSIM and LPIPS indicators. The prediction effect of t+5 frame is slightly lower than SwinLSTM, but it still has better results than other models. In short-term prediction tasks, TGDVFN has a strong ability to capture motion trends, while also maintaining a high image quality, and has obvious advantages in the comparison of visual effects and evaluation indicators. It can be seen that TGDVFN in this embodiment has a more obvious advantage in the short-term prediction task of cloud clusters.

[0099] Table 2

[0100]

[0101] In addition, TGDVFN is compared with MotionRNN, which performs well in the short-term precipitation prediction task, on the Shanghai radar echo dataset. Figure 7As can be seen in the figure, when the MotionRNN model uses three frames as input, the details of the prediction results are worse than those of the TGDVFN model which only uses two frames as input. Compared with MotionRNN, TGDVFN also has better structural similarity, more detailed features, and is more similar to the real radar echo map.

[0102] In the model proposed in this embodiment, the introduction of the ViT-M module is a key innovation. It enables the model to effectively collect and integrate information from the entire data set when processing the prediction task of the image sequence. This global information aggregation capability greatly enhances the efficiency of the model in extracting useful features from complex data, so that when predicting the t+1 frame image, the model can use richer and more relevant information to generate accurate prediction results. At the same time, the prediction results are indirectly generated with the help of voxel flow, so that the images generated by the model have better image similarity. The design of the scaling factor enables the model to have better feature capture capabilities for large-scale and small-scale motions. The model also has excellent capabilities in small-scale motion prediction. Finally, the discriminator module of PatchGAN helps the model to further improve the correctness of each patch of the image and grasp local details. These designs enable the TGDVFN proposed in this embodiment to show relatively excellent performance in both the self-built data set and the Shanghai radar echo data set.

[0103] Ablation experiment and analysis: At the beginning of training, the weight coefficient λ of the generator loss function is set to 0.5. In order to explore the impact of the λ setting size on the model results, this example conducts the impact of different weight coefficients on the model prediction results in HiwDataset, as shown in Table 3.

[0104] Table 3

[0105]

[0106] Through this ablation experiment, it was found that the model works best when the weight coefficient λ of the generator loss function is set to 0.4. In the overall proportion of the model, it can be observed that the change of the discriminator's weight will affect the model. Specifically, when λ is set to around 0.5, the model maintains a relatively good effect. When the weight coefficient is reduced or made too large, the model's effect declines to varying degrees. This result verifies the design structure of TGDVFN's GAN network model, which has a positive impact on the model when the weight coefficient is set reasonably.

[0107] This embodiment also experiments on the HiwDataset data for quantitative results with different scaling factor settings, and the evaluation index is MS-SSIM, which specifically includes:

[0108] ·"[1]":[1,1,1,1,1]

[0109] ·"[2]":[2,2,2,2,2]

[0110] ·"[4]":[4,4,4,4,4]

[0111] ·"[1,4]":[1,1,4,4,4]

[0112] ·"[4,1]":[4,4,4,1,1]

[0113] ·"[2,4]":[2,2,4,4,4]

[0114] ·"[4,2]":[4,4,4,2,2]

[0115] ·"[1,2,4]":[1,2,2,4,4]

[0116] ·"[4,2,1]":[4,4,2,2,1]

[0117] As shown in Table 4, the prediction effect of TGDVFN[4,2,1] is better than other settings, and it performs better in long-term prediction tasks. This proves that the model first extracts the features of large-scale motion, and then extracts specific small-scale motion or detail information, which is conducive to the grasp of global spatiotemporal information. The large-scale motion feature information contains the low-dimensional information of small-scale motion. This scaling factor sequence design from large to small helps the model better capture large motion while enriching the low-dimensional information of small-scale motion, significantly enhancing the model's ability to capture multi-scale motion and detail information.

[0118] Table 4

[0119]

[0120]

[0121] This embodiment proposes a DVF-based cloud short-term motion trend prediction model TGDVFN, which adopts an advanced hybrid feature extraction strategy, uses a convolutional neural network (CNN) to extract global spatial information, and ViT-M based on VisionTransformer to extract cloud motion information. This hybrid feature extraction strategy, as the core part of the generator design, enables the generator to effectively capture the detailed changes of clouds and their motion trends. In order to predict the motion trend of cloud images with higher quality, this embodiment selects PatchGAN as the discriminator. The design feature of PatchGAN is that it can distinguish local areas of the image, which makes the model more refined when processing image details, further improving the accuracy and credibility of the prediction. In solving the problem of large-scale motion prediction of clouds, by introducing a sequence of scaling factors, the model can capture the details of cloud motion at different scales, greatly enhancing the perception of large-scale dynamic changes. In the design of the loss function, the L1 loss calculated on each pair of images is represented by a Laplacian pyramid to collect information at each layer. A large number of experiments have shown that TGDVFN has achieved remarkable performance in the task of predicting cloud movement trends, all of which prove the effectiveness of the method proposed in this embodiment.

[0122] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for predicting short-term cloud movement trends, characterized in that: include: Obtain satellite cloud image sequence data sets and build a cloud short-term motion trend prediction model based on deep voxel flow and Vision Transformer; Based on the satellite cloud image sequence data set, a cloud cluster short-term motion trend prediction model is trained to obtain a target prediction model; The short-term movement trend of the cloud cluster is predicted by the target prediction model to obtain a prediction result; The cloud cluster short-term motion trend prediction model also includes five TGDVF sub-networks in series. The i-th cloud cluster short-term motion trend prediction model is expressed as By combining the two frames I t-1 and I t Synthesized Frames and the estimated voxel flow As input, learn to predict the target voxel flow At the same time, the last frame synthesized by the TGDVF sub-network Will be used as input to the discriminator; The process of predicting the short-term movement trend of clouds by using the target prediction model includes: For the cloud to be predicted, the predicted image is indirectly generated by optical flow mapping; A scaling factor sequence is added to the target prediction model, and two channels are designed to extract the overall spatial features of the cloud image and the motion features of the cloud clusters using CNN and ViT-M respectively through a pyramid optical flow estimation method. Using each layer of the target prediction model represented by the Laplacian pyramid, outputting the sum of the reconstruction losses as the loss function, and the PatchGAN discriminator to form a GAN network model, discriminating the local area in the predicted image based on the GAN network model to obtain the local change; Obtaining a prediction result of the short-term movement trend of the cloud group to be predicted according to the local changes; The process of adding a scaling factor sequence to the target prediction model includes: Design scaling factor s according to the motion scale differences in different regions i , by scaling factor s i Perform downsampling; The scaling factor s i Reduce image resolution and expand receptive field by downsampling; Wherein, the scaling factor s i For a decreasing sequence: I i =Downsample(I original ,S i ) F i =OpticalFlowEstimation(I i ) F i，refined =Refine(F i-1 ,S i ) Among them, I i is the downsampled image at the current level, s i is the scaling factor for the current level, F i is the estimated optical flow field at the current level, F i，refined It is the optical flow field refined by the optical flow field of the previous level; The process of optical flow estimation by pyramid optical flow estimation method includes: Starting from applying the largest scaling factor, downsample the original image to obtain a small image version; Performing optical flow estimation based on the small image version to obtain a preliminary estimate of the optical flow field; Then the scaling factor is gradually reduced, the resolution of the image is gradually increased, and the optical flow field is refined and adjusted accordingly. The previously estimated optical flow field at each step is used as the initial condition for the estimation of the next resolution level to obtain the final estimate of the optical flow field. The process of designing two channels to use CNN and ViT-M to extract the overall spatial features of the cloud image and the cloud movement features includes: A dual-branch network structure is constructed by using a pyramid optical flow estimation method. One channel is performed at high resolution to extract features from the image at the original resolution to obtain spatial information. The other channel first uses bilinear interpolation to scale the image according to the scaling factor. Subsequently, ViT-M is used to extract the motion features of the local cloud clusters after segmentation. After that, the feature map is restored to the same dimension as the other channel through upsampling and then merged. Finally, the voxel flow is generated through deconvolution. It means that the output of the i-th TGDVF has the following relationship: in, and The initial value of is set to zero; The ViT-M modifies ViT by setting the moving step of the rectangular area to half of the rectangular frame, and then selecting an area and translating the area by 8 units in sequence; for the features of the overlapping extracted areas, the weighted average method is used to integrate them to obtain ViT-M.

2. The method for predicting short-term cloud movement trends according to claim 1, characterized in that: The loss function is the reconstruction loss output for each block The sum of is expressed as:

3. The method for predicting short-term cloud movement trends according to claim 1, characterized in that: The PatchGAN discriminator includes several convolutional layers. The activation function of each convolutional layer adopts the LeakyRelu activation function. The final convolution result is input into the fully connected layer. The output layer adopts the sigmoid activation function. The discriminator converts the I generated by the generator into t+1 As input, it distinguishes true from false, with 1 representing "true" and 0 representing "false". When the model converges, the discriminator will judge the predicted frame as 0 and the real frame as 1.

4. The method for predicting short-term cloud movement trends according to claim 1, characterized in that: The PatchGAN discriminator is used for each N×N patch in the image. The PatchGAN discriminator tries to determine whether the patch is real or fake; Where I is the input image, I m is the mth patch in the image, M is the total number of patches in the image, and D patch It is the judgment function of the PatchGAN discriminator for a single patch.

Images