A satellite cloud image sequence prediction method based on improved SimVP

Abstract

The application discloses a satellite cloud image sequence prediction method based on improved SimVP. It relates to the field of deep learning. The specific steps are as follows: (1) preprocessing satellite cloud image sequence data and dividing training set and test set; (2) based on the existing SimVP architecture, adding a multi-head attention mechanism module and a local residual connection in the encoder and the decoder, using a Poolformer network as a translator, and adding a generator of an SRGAN network at the end of the network; (3) combining a motion perception loss function and an L2 loss function to supervise the model training; (4) predicting satellite cloud image sequence images. The application introduces a multi-head attention mechanism in a multi-layer convolutional neural network, which can well extract high-dimensional features of satellite cloud image sequence data, so that the model can learn the timing correlation while maintaining the spatial information of the sequence; and a generator is introduced to improve the resolution of the predicted cloud image, solving the problem of feature loss caused by blurred predicted images; at the same time, the motion perception loss function is combined with the traditional L2 loss function to supervise the model training, which can further obtain the motion information between adjacent frames of the satellite cloud image sequence. The application is not only suitable for all satellite cloud image sequences, but also can be applied to sequence image prediction in some other complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of deep learning, specifically relating to a satellite cloud image sequence prediction method based on improved SimVP. Background Technology

[0002] Satellite cloud images are images of Earth's cloud cover and surface features observed by meteorological satellites. They reflect cloud formation, dissipation, movement, and changes, and are a crucial data source for weather forecasting and meteorological analysis. Timely and accurate predictions of future satellite cloud image sequences are essential for identifying different weather systems, estimating their intensity and development trends, improving the accuracy and timeliness of weather forecasts, and preventing and mitigating meteorological disasters.

[0003] Traditional cloud image sequence prediction is generally based on block matching and optical flow methods. However, these two methods are mostly linear predictions, failing to consider the numerous nonlinear changes that occur during atmospheric motion. They ignore processes such as cloud growth, dissipation, splitting, and merging, often making it difficult to handle complex cloud field structures and multi-scale cloud movements. Their predictive capabilities are limited, and they are prone to forecast bias and cumulative errors. Furthermore, traditional cloud image prediction methods involve a high degree of human intervention throughout the entire prediction process, resulting in low prediction efficiency. In recent years, the development of deep learning has provided new ideas for cloud image sequence prediction. Cloud image sequence prediction is essentially a spatiotemporal sequence prediction problem. By using deep learning-based spatiotemporal sequence prediction models, the potential patterns of cloud movement and change can be learned from historical data, effectively addressing the aforementioned problems of traditional prediction methods. Common deep learning-based spatiotemporal sequence prediction models include recurrent neural networks such as ConvLSTM and PredRNN, and convolutional neural networks (CNNs) such as SimVP and TAU. These prediction models have shown good performance in simple video prediction tasks. In particular, SimVP is a video prediction network entirely built on CNN, providing a scalable framework for spatiotemporal sequence prediction. This network consists of three parts: an encoder, a translator, and a decoder. These three parts are responsible for extracting spatial features, learning spatiotemporal evolution patterns, and integrating spatiotemporal information to reconstruct real frames, respectively. Because it has no recurrent architecture, does not add other complex modules, and does not require the introduction of any additional techniques or strategies, this network achieves efficient prediction while maintaining a simple network structure. However, satellite cloud image prediction is more challenging than general video prediction tasks. First, unlike typical video frames which emphasize object outlines, cloud images focus more on cloud morphology, size, density, and altitude. These characteristics often affect light propagation, leading to blurred predicted images. Second, video prediction only considers motion information and color changes between frames, while cloud changes are influenced by many complex meteorological factors, potentially resulting in unclear cloud edges or abnormal changes in cloud movement trajectories, thus increasing the number and complexity of cloud image features. Therefore, directly using general video prediction networks to predict satellite cloud image sequences will lead to inaccurate and blurry prediction results.

[0004] To address the aforementioned issues, this invention proposes a satellite cloud image sequence prediction method based on an improved SimVP. Considering the characteristics of cloud images, this method improves the prediction model on top of SimVP. First, to extract more cloud image features and enable the model to retain spatial information of the sequence while learning temporal correlations, this method introduces a multi-head attention mechanism and local residual connections in the encoder and decoder. Second, to address the feature loss problem caused by blurred predicted images, a generator is introduced to improve the resolution of the predicted cloud image. Finally, this invention also proposes a motion-aware loss function to obtain motion information between adjacent frames of the satellite cloud image sequence and combines it with the traditional L2 loss function to supervise model training. Summary of the Invention

[0005] The technical problem to be solved by this invention is to overcome the shortcomings of traditional cloud image sequence prediction and spatiotemporal sequence prediction methods, and to propose a satellite cloud image sequence prediction method based on improved SimVP, which extracts more cloud image features, obtains potential cloud motion information, and adopts strategies such as enhancing cloud image resolution to obtain better satellite cloud image sequence prediction performance.

[0006] The technical solution adopted in this invention is: to propose a satellite cloud image sequence prediction method based on improved SimVP, which is implemented according to the following steps:

[0007] Step 1: Obtain satellite cloud image sequence data, preprocess the data, and divide it into training and test sets;

[0008] Step 2: Build a satellite cloud image sequence prediction network based on the existing SimVP encoder-translator-decoder architecture. First, add a multi-head attention mechanism module and local residual connections to the encoder and decoder; second, replace the original translator with a Poolformer network; finally, train the SRGAN network using real cloud image data, and extract the fixed parameters of the generator after training and connect them separately to the back end of the prediction network decoder.

[0009] Step 3: Train the model based on the training set obtained in Step 1, adjust parameters such as batch size, learning rate, optimizer, number of epochs, and combine motion-aware loss function and L2 loss function to supervise the model training.

[0010] Step 4: Input the test set satellite cloud image sequence obtained in Step 1 into the trained prediction network to obtain the predicted cloud image sequence, and evaluate the prediction results using the corresponding evaluation metrics.

[0011] The specific steps for step 2 above are as follows:

[0012] (1) Constructing the basic encoder structure. In order to encode the input high-dimensional satellite cloud image past frames into a low-dimensional latent space, we used 6 sets of convolutional layers in the encoder. The hidden representation is shown below: z i =σ(Norm2d(Conv2d(z) i-1 ))), 1≤i≤6

[0013] Where σ is the nonlinear activation function SiLU, Norm2d is a normalization layer Groupnorm, and Conv2d is the convolutional layer operation. z0 represents the tensor input to the convolutional layer. The stride of the convolutional layer is set to 1, and then a downsampling operation is performed once for every two convolutional layers, with a stride of 2.

[0014] (2) Constructing the basic decoder structure. In the decoder, to decode the information in the latent space into a predicted future satellite cloud image, we also use 6 sets of deconvolution layers, and the hidden representation is as follows: W k =σ(Norm2d(unConv2d(w k-1 ))), 1≤k≤6

[0015] Where σ is the nonlinear activation function SiLU, Norm2d is a normalization layer Groupnorm, and unConv2d is the deconvolution layer. w0 represents the tensor input to the deconvolution layer. The stride of the deconvolution layer is set to 1, and then an upsampling operation is performed once for every two deconvolution layers using pixclshuffle, with a stride of 2.

[0016] (3) A multi-head attention mechanism is added before the convolution and deconvolution operations inside the already constructed basic encoder and decoder. The principle is to map the original features of the input satellite cloud image to different feature spaces query, key, and value, and then pack them into matrices Q, K, and V respectively. In order to better learn the spatiotemporal dependencies of the satellite cloud image sequence and fully obtain the structural features of the cloud image, we linearly project Q, K, and V according to different weights to obtain Q', K', and V', and calculate the attention function for the different projection versions in turn. Finally, we concatenate each generated value and project it again to obtain the final value. The specific calculation process is as follows: MultiHead(Q,K,V)=Concat(head1,head2,…,head h W O head i =Attention(Q′) i , K′ i , V′ i i = 1, 2, ..., h

[0017] in W O These represent the corresponding matrix coefficients.

[0018] Then, a local residual connection is added between the input and output of the multi-head attention mechanism, and a residual connection is added between the first layer of the encoder and the last layer of the decoder to preserve spatial features.

[0019] (4) A Poolformer network is used as the translator of the prediction network to learn the spatial dependencies and temporal variations of satellite cloud image sequences from the latent space. The output tensor of the encoder is used as the input of the Poolformer, and multiple sequence frames are stacked along the time axis. The Poolformer is then used to learn from the features of the stacked multi-frames to capture the inherent temporal evolution of the satellite cloud image sequence data. Finally, the output is used as the input of the decoder.

[0020] (5) Train the SRGAN network using the satellite cloud image dataset from step 1. SRGAN consists of a generator and a discriminator. The generator is responsible for converting low-resolution images into high-resolution images, while the discriminator is responsible for distinguishing between the high-resolution images generated by the generator and the real high-resolution images. During training, satellite cloud images are downsampled into low-resolution images and input into the generator, while real satellite images are directly input into the discriminator. The training process uses an alternating optimization approach. First, the discriminator's network parameters are fixed, and the generator's network parameters are updated to minimize the generator's loss function. Here, the generator uses content loss and adversarial loss. The purpose of content loss is to ensure that the generated high-resolution images are as close as possible to the real images, while adversarial loss encourages the generator to generate more realistic images through adversarial interaction with the discriminator network. Then, the generator's network parameters are fixed, and the discriminator's network parameters are updated to minimize the discriminator's loss function. Here, the discriminator uses adversarial loss and perceptual loss. This alternating optimization process continues until the model converges. Finally, after training, the generator's parameters are fixed and ported to the backend of the satellite cloud image prediction network.

[0021] The loss function used in step 3 above is shown below: L=L2+αLm

[0022] Where Y i and represents the true value and predicted value at the i-th time step, respectively, and τ represents the total future time step. Lm is the motion-aware loss function. COF is a function used to calculate dense optical flow, which is used here to calculate the motion changes between two adjacent frames. This function first takes two images as input, calculates the gradients of the images in the x and y directions, then constructs a Gaussian pyramid on the images to form multiple scale levels, calculates the optical flow field at each scale separately, and finally merges them to obtain the final optical flow field. α is a hyperparameter that balances the L2 loss and Lm loss during training, and here the default a = 0.5;

[0023] The specific steps for predicting satellite cloud image sequences using the test set in step 4 above are as follows:

[0024] Input satellite cloud image sequence X∈R into the prediction network T×C×H×W Where T represents the total number of images in the input satellite cloud image sequence, C represents the number of channels in each image, and H and W represent the height and width of each image, respectively. A dynamic sequence is formed based on the changes X over T time steps. The satellite cloud image sequence prediction predicts the future T' frames based on the given T frames of cloud images, and its modeling formula is as follows:

[0025] Where Θ represents a series of parameters that need to be optimized.

[0026] In step 4, standard evaluation indicators such as MSE, MAE, SSIM, and PSNR can be used to evaluate the predicted satellite cloud images. For satellite cloud images with different cloud category labels, non-standard indicators commonly used in meteorology, such as frequency deviation (FB), Brier score (BS), and Brier skill score (BSS), can also be used to evaluate the prediction results.

[0027] The calculation method for the standard evaluation indicators is as follows:

[0028] Where Y and Let w and h represent the actual value and the predicted value, respectively, and let μ represent the width and height of each frame. Y and Representing Y and The average value, σ Y and Representing Y and standard deviation Representing Y and The covariance, c1 and c2 are constants, I MAX This represents the maximum pixel value of the image. Here, because the grayscale values ​​are normalized to [0, 255], I... MAX =255;

[0029] FB is the ratio of the number of pixels predicted as clouds by the model to the actual number of cloud pixels. An FB greater than 1 (or less than 1) indicates that the prediction model tends to overestimate (or underestimate) the event.

[0030] BS is a way to measure the difference between model predictions and actual values. Its value is between 0 and 1. The closer the score is to 0 (1), the better (poor) the prediction result is and the better (poor) the calibration is.

[0031] BSS is calculated using the MSE predicted by a given model relative to the baseline. BSS > 0 (BSS < 0) indicates that the model's prediction is better (bad).

[0032] The calculation methods for BS and BSS are as follows:

[0033] Where M represents the number of cloud categories, N represents the number of frames predicted, and P t，k Let y represent the predicted probability of the k-th type of cloud at time t. t，k Indicates true label 0 or 1, BS model BS represents the BS value obtained from the prediction model. persistence This represents the baseline predicted BS value.

[0034] The beneficial effects of this invention are:

[0035] A satellite cloud image sequence prediction method based on improved SimVP is proposed, which is more efficient than recurrent neural prediction networks. Furthermore, multi-head attention and local residual connections are introduced into the encoder and decoder, enabling the prediction network to learn the spatiotemporal dependencies and more spatial features of the satellite cloud image sequence. Simultaneously, the generation capability of the generator is utilized to improve the resolution of the predicted cloud image and preserve more detailed features. In addition, the proposed motion-aware loss function can be used as a supervision signal to supervise the model in learning high-level features of motion changes between cloud image sequences. Attached image description:

[0036] Figure 1 This is a diagram of the internal structure of the present invention;

[0037] Figure 2 This is a diagram of the internal structure of SRGAN;

[0038] Figure 3 This is a comparison chart of the prediction results of different prediction methods on the FY-1A dataset;

[0039] Figure 4 This is a graph showing the differences in prediction results for the last frame of different prediction methods on the FY-4A dataset.

[0040] Figure 5 This is a graph showing the differences in prediction results for the last frame of the CloudCast dataset using different prediction methods. Detailed Implementation

[0041] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The following examples are used to illustrate the present invention, but are not intended to limit the scope of protection of the present invention.

[0042] Example 1:

[0043] This example demonstrates a satellite cloud image sequence prediction experiment on the FY-4A dataset. The FY-4A dataset consists of satellite cloud image sequence data from 2018 to 2020, with a 30-minute time interval between adjacent frames.

[0044] The present invention is used to predict satellite cloud image sequences on the FY-4A dataset. The specific implementation steps are as follows:

[0045] In Example Step 1, select data from channel 11 of FY-4A, visualize it, and normalize its grayscale values ​​to [0, 255]. Select the central 128×128 region of each image as the data used in the experiment. Use 75% of the dataset as the training set and the remaining 25% as the test set. Each cloud image sequence consists of 16 consecutive cloud image images. Use the first 8 images as input to the prediction network, i.e., use satellite cloud images from the past 4 hours to predict the cloud image sequence for the next 4 hours.

[0046] Example Step 2, Building the Prediction Model:

[0047] First, a basic prediction network with an encoder-translator-decoder structure is constructed. Then, an SRGAN network is trained using the FY-4A dataset. During training, the Batch Normalization (BN) layer of the original SRGAN network is removed to reduce computational complexity and improve the stability of the network's generalization ability. After training, the obtained generator weights are fixedly connected to the end of the basic prediction network to form the final prediction network, such as... Figure 1 As shown.

[0048] Example step 3: Train the model based on the training set obtained in step 1:

[0049] Using the Adam optimizer and a OneCycle learning rate variation strategy with an initial learning rate of 2e-4, the training batch size for all prediction methods was set to 16, except for E3DLSTM, which was set to 4 due to excessive memory cost. A total of 200 epochs were trained.

[0050] Step 4: Input the satellite cloud image sequence obtained in Step 1 into the trained prediction network to obtain the predicted cloud image sequence, and evaluate the prediction results using appropriate evaluation metrics. The test results of different methods on the FY-4A dataset are shown in Table 1: Table 1 Experimental results of the FY-4A dataset

[0051] As shown in Table 1, the evaluation metrics of the method of this invention are superior to other prediction methods. The input cloud map sequence, the actual cloud map sequence, and the cloud map sequences predicted by different methods are shown below. Figure 2 As shown in the figure, the prediction results are satisfactory in the short term, but over time, the method of this invention is generally superior to other methods in predicting the shape and location of clouds.

[0052] Figure 3The differences between the predicted cloud image in frame 8 and the actual value are shown for each prediction method. The method of this invention outperforms other methods for predicting different types of clouds, and has the smallest error among all prediction methods.

[0053] Example 2:

[0054] This example demonstrates a prediction experiment conducted on the CloudCast dataset. This dataset was created in 2017. 1 The cloud images consisted of 70,080 samples collected from January to December 2018, with adjacent samples spaced 15 minutes apart. Based on cloud height, they were categorized into four types: cloudless, low-cloud, mid-cloud, and high-cloud. The cloud images used in the experiment were created by overlaying cloud-marked images and satellite maps onto corresponding geographic coordinates (longitude and latitude). The remaining operational steps were similar to those in Example 1, with the cloud images cropped to 128×128 pixels, 75% used as the training set, and the remaining 25% used as the test set.

[0055] The training batch size was set to 4, with other settings the same as in Example 1. Furthermore, since the cloud images in this dataset include four different types of cloud labels, we can use FB, BS, and BSS to evaluate the prediction results. The baseline prediction used here is a commonly used and recommended benchmark model in cloud assessment research. In this baseline prediction, satellite images labeled with clouds are delayed by 15 minutes and then replicated for the same 16 frames in the future. This assumption only applies to cases with limited cloud movement and is not effective under dynamic weather conditions.

[0056] The test results of different methods on the CloudCast dataset are shown in Table 2: Table 2 Experimental Results of CloudCast Dataset Benchmark forecast 1.00 0.18 NA 0.55 <![CDATA[AE - ConvLSTM]]> 0.96 0.16 0.01 0.66 MD-GANS2 0.95 0.16 0.07 0.60 TVL1 1.10 0.18 0.02 0.58 SimVP 1.12 0.15 0.07 0.68 Method of the present invention 1.09 0.12 0.08 0.71

[0057] As shown in Table 2, the method of this invention overestimates the cloud image sequence prediction. Looking at the three metrics—BS, BSS, and SSIM—the method of this invention outperforms other methods in predicting the CloudCast dataset.

[0058] Figure 4 The comparison between SimVP's and the proposed method's predictions and actual values ​​for the next five time points on the CloudCast dataset is presented. Figure 4 (b) The blue part represents the difference between the predicted value and the actual value. It can be seen that the cloud map predicted by the method of the present invention has a smaller difference.

[0059] The test results of the above embodiments demonstrate the effectiveness and superiority of the present invention in satellite cloud image sequence prediction.

Claims

1. A satellite cloud image sequence prediction method based on improved SimVP, characterized in that, The steps include the following: Step 1: Obtain satellite cloud image sequence data, preprocess the data, and divide it into training and test sets; Step 2: Build a satellite cloud image sequence prediction network based on the existing SimVP encoder-translator-decoder architecture. First, add a multi-head attention mechanism module and local residual connections to the encoder and decoder. Second, replace the original translator with a Poolformer network. Finally, train the SRGAN network using real cloud image data, and extract the fixed parameters of the generator after training and connect them separately to the back end of the prediction network decoder. Step 3: Train the model based on the training set obtained in Step 1, adjust parameters such as batch size, learning rate, optimizer, number of epochs, and combine motion-aware loss function and L2 loss function to supervise the model training. Step 4: Input the test set satellite cloud image sequence obtained in Step 1 into the trained prediction network to obtain the predicted cloud image sequence, and evaluate the prediction results using the corresponding evaluation metrics. The specific steps of step 2 are as follows: (1) Constructing the basic encoder structure: In order to encode the input high-dimensional satellite cloud image past frames into the low-dimensional latent space, we used 6 sets of convolutional layers. The hidden representation is shown below: in It is a nonlinear activation function SiLU. It is a normalized layer, Groupnorm. That is, convolutional layer operations. This represents the tensor input to the convolutional layer. The stride of the convolutional layer is set to 1. Then, a downsampling operation is performed once for every two convolutional layers, with a stride of 2. (2) Constructing the basic decoder structure, in order to decode the information in the latent space into the predicted future satellite cloud image, we also use 6 sets of deconvolution layers, and the hidden representation is as follows: in It is a nonlinear activation function SiLU. It is a normalized layer, Groupnorm. That is, deconvolution layer. This represents the tensor input to the deconvolution layer. The stride of the deconvolution layer is set to 1. Then, for every two deconvolution layers, an upsampling operation is performed using pixel shuffle with a stride of 2. (3) Before the convolution and deconvolution operations inside the built basic encoder and decoder, a multi-head attention mechanism is added. The principle is to map the original features of the input satellite cloud image to different feature spaces query, key, value, and then pack them into matrices Q, K, and V respectively. In order to better learn the spatiotemporal dependencies of the satellite cloud image sequence and fully obtain the structural features of the cloud image, we linearly project Q, K, and V according to different weights to obtain Q', K', and V'. Then, we calculate the attention function for the different projection versions in turn, and finally connect each generated value and project it again to obtain the final value. The specific calculation process is as follows: in These represent the corresponding matrix coefficients; Then, a local residual connection is added between the input and output of the multi-head attention mechanism, and a residual connection is added between the first layer of the encoder and the last layer of the decoder to preserve spatial features; (4) The Poolformer network is used as the translator of the prediction network to learn the spatial dependence and temporal variation of the satellite cloud image sequence from the latent space. The output tensor of the encoder is used as the input of the Poolformer, and multiple sequence frames are stacked along the time axis. Then, the Poolformer is used to learn from the features of the stacked multi-frames to capture the inherent temporal evolution law inside the satellite cloud image sequence data. Finally, the output result is used as the input of the decoder. (5) Use the satellite cloud image dataset from step 1 to train the SRGAN network: SRGAN consists of a generator and a discriminator. The generator is responsible for converting low-resolution images into high-resolution images, and the discriminator is responsible for distinguishing between the high-resolution images generated by the generator and the real high-resolution images. During training, the satellite cloud images are downsampled into low-resolution images and input into the generator, while the real satellite images are directly input into the discriminator. The training process is carried out in an alternating optimization manner. First, the network parameters of the discriminator are fixed, and the network parameters of the generator are updated to minimize the loss function of the generator. The loss function used by the generator here includes content loss and adversarial loss. The purpose of content loss is to ensure that the generated high-resolution images are as close as possible to the real images. The adversarial loss is to encourage the generator to generate more realistic images by adversarially opposing the discriminator network. Then, the network parameters of the generator are fixed, and the network parameters of the discriminator are updated to minimize the loss function of the discriminator. The loss function used by the discriminator here is adversarial loss and perceptual loss. The entire alternating optimization process continues until the model converges. Finally, after training is completed, the parameters of the generator are fixed and transferred to the back end of the satellite cloud image prediction network. In step 3, the loss function used is: in and Let these represent the actual value and the predicted value at the i-th time step, respectively. Representing the total future time step, Lm is the motion-aware loss function, and COF is the function used to calculate dense optical flow. Here, it is used to calculate the motion changes between two adjacent frames. This function first takes two images as input, calculates the gradients of the images in the x and y directions, then constructs a Gaussian pyramid on the images to form multiple scale levels, calculates the optical flow field at each scale separately, and finally merges them to obtain the final optical flow field. These are the hyperparameters that balance the L2 loss and Lm loss during training; they are set to default values ​​here. =0.

5.

2. The satellite cloud image sequence prediction method based on improved SimVP according to claim 1, characterized in that, The specific steps for predicting the satellite cloud image sequence in step 4 are as follows: Input satellite cloud image sequence X∈R into the prediction network T×C×H×W Where T represents the total number of images in the input satellite cloud image sequence, C represents the number of channels in each image, and H and W represent the height and width of each image, respectively. A dynamic sequence is formed based on the changes X over T time steps. The satellite cloud image sequence prediction predicts the future T' frames based on the given T frames of cloud images. The modeling formula is as follows: ( , ) in This represents a series of parameters that need to be optimized. The relevant evaluation is conducted using metrics such as MSE, MAE, SSIM, and PSNR, as detailed below: in Let w and h represent the actual value and the predicted value, respectively, and w and h represent the width and height of each frame, respectively. They represent The average value, and They represent standard deviation express covariance, and It is a constant. This represents the maximum pixel value of the image. Here, because the grayscale values ​​are normalized to [0, 255], therefore... =255; For satellite cloud images with different cloud category labels, non-standard indicators commonly used in meteorology, such as frequency deviation (FB), Brier score (BS), and Brier skill score (BSS), can also be used to evaluate the forecast results.

Images