Satellite cloud picture prediction method based on implicit autoregression space-time channel aggregation strategy

Abstract

The invention provides a satellite cloud picture prediction method based on an implicit autoregression space-time channel aggregation strategy, and the method comprises the steps: introducing an implicit autoregression codec, endowing a model with a certain recursive characteristic through employing an implicit autoregression structure, so as to capture the space-time correlation information in a sequence; error accumulation caused by recursion is reduced by using a mask and a real mapping module, and the robustness of the model is enhanced; a predictor is designed for the problems of instable cloud motion, nonlinearity and the like, the module can effectively aggregate multi-order space-time interaction information, and adaptive channel redistribution is carried out on the multi-order space-time interaction information to reduce feature redundancy. Experiments are carried out on a MovingMNIST data set and an FY-4A satellite cloud picture data set, the experimental result shows that the defects of MIMO and SISO models are effectively overcome, the prediction precision is higher than that of other models, and the reliability and effectiveness of the method in the field of cloud picture prediction are embodied.

Description

Technical Field

[0001] The present invention belongs to the field of spatiotemporal sequence prediction, and in particular relates to a satellite cloud image prediction method based on an implicit autoregressive spatiotemporal channel aggregation strategy. Background Art

[0002] Satellite cloud image prediction is a method of using cloud image data obtained by satellites to predict weather and climate change. The essence of cloud image prediction is spatiotemporal sequence prediction. The current spatiotemporal prediction algorithms based on deep learning can be divided into three types of frameworks, namely single-input single-output framework (Single-In-Single-Out, SISO), multiple-input multiple-output framework (Multiple-In-Multiple-Out, MIMO) and multiple-input single-output framework (Multiple-In- Single-Out, MISO).

[0003] The SISO framework prediction model mainly uses recurrent neural networks and their variants, long short-term memory networks (LSTM) and gated recurrent units (GRU) to give the model recursive characteristics and realize sequence prediction.

[0004] MIMO-type prediction algorithms input the entire sequence of frames into the encoder, and after the predictor conversion, decode all frames at the same time, breaking the error accumulation chain. MIMO-type models do not rely on the prediction method of the previous frame, which solves the problem of error accumulation. However, this type of model does not contain autoregressive characteristics, and there is a lack of temporal and spatial correlation between the predicted frames.

[0005] In the field of cloud map prediction based on deep learning, neural networks are generally used to predict the next few frames of time series cloud maps based on the previously observed frames of time series cloud maps.

[0006] Single-input single-output prediction framework: The prediction model based on the SISO framework is essentially a recursive model, that is, by learning the hidden state of historical sequence information, the next frame is generated based on the current frame. Therefore, when a small error occurs in the prediction frame in the early stage of the sequence, a butterfly effect will occur. As the generated frame lengthens, the quality and accuracy of the image frame will drop rapidly.

[0007] Multiple-input multiple-output prediction framework: The MIMO framework-based model inputs all sequence frames into the encoder, and after the predictor learns the spatiotemporal information, the decoder is used to decode all frames at the same time. Compared with the SISO framework, the prediction results of this type of framework model do not depend on the previous prediction results, and there will be no error accumulation. However, it should be noted that the MIMO model is not recursive, and the method of decoding and outputting all frames at once makes the frames predicted by this type of model lack correlation. Summary of the invention

[0008] Purpose of the invention: To solve the problem of image correlation loss in MIMO models and excessive error accumulation in SISO models, the present invention provides a satellite cloud image prediction method based on implicit autoregressive spatiotemporal channel aggregation strategy, comprising the following steps:

[0009] Step 1, establish an IASCACP network model based on implicit autoregressive spatiotemporal channel aggregation strategy;

[0010] Step 2, executing an implicit autoregressive training strategy on the IASCACP network model to obtain a trained IASCACP network model;

[0011] Step 3: Use the trained IASCACP network model to predict satellite cloud images.

[0012] In step 1, the IASCACP network model includes an encoder, a predictor and a decoder;

[0013] The length of the sequence image at time t is n Input encoder, ,in , , and Respectively represent the frame length, number of channels, height and width of the sequence image; R represents all time series images;

[0014] Assume the loop frame sequence at time t is , then the predicted frame at time t is ;

[0015] The prediction process of the IASCACP network model is expressed as:

[0016] (1)

[0017] in , and denote the encoder, predictor and decoder respectively; is the position encoding function, Mask is the masker, and Concat is the stacking operation.

[0018] In step 1, the decoder is an implicit autoregressive codec, including a position encoding module PE, a spatiotemporal refinement module STR and a mask module; the decoder also includes a prediction frame output port, and a real mapping module TM is introduced at the prediction frame output port, and the real mapping module TM is used to project a feature vector on the prediction frame.

[0019] In step 1, the frame sequence processed in the IASCACP network model includes an observation frame sequence, a loop frame sequence and a mask frame sequence;

[0020] The observation frame sequence is the result obtained by passing the input sequence through the encoder, and the step length of the observation frame sequence remains unchanged as time increases, and always has the same encoding value;

[0021] The loop frame sequence is initially filled with zero tensors. As the time step advances, the predicted frames are gradually output and filled into the loop frame sequence after passing through the real mapping module TM.

[0022] The loop frame sequence is then passed through a masker Mask to obtain a mask frame sequence;

[0023] The mask frame sequence and the observation frame sequence are stacked to obtain a stacked sequence, and the position encoding module PE is used to model the position relationship in the stacked sequence and enhance the spatiotemporal correlation between sequences (the sequence is weighted by position weights to enhance the spatiotemporal correlation);

[0024] The mask frame sequence and the observation frame sequence are stacked and combined with the data generated by the position encoding module PE, and then output to the predictor to learn the spatiotemporal trend changes, and then decoded by the decoder to obtain the current predicted frame.

[0025] In step 1, the spatiotemporal refinement module STR refines the decoded frame; while calculating the loss of the predicted frame and the label frame, the predicted frame is converted into a loop frame sequence through the real mapping module TM, and the loop frame sequence is then masked and used together with the input frame to predict the next frame;

[0026] In order to reduce the training cost during model training, the real future frames are filled into the loop frame sequence in the training phase, and the predicted frames are filled into the loop frame sequence in the verification phase.

[0027] In step 1, the mask module is used to randomly mask redundant information in the loop frame sequence during the training phase to provide incomplete contextual spatiotemporal sequence information.

[0028] Step 2 includes: setting the input frame to , the real future frame is , after passing through the implicit autoregressive encoder-decoder, we get:

[0029] (2)

[0030] in, Predict the frame length for the target, is the observation frame sequence, is the loop frame sequence, is the position encoder in the encoding module PE, is the i-th observation frame, is the i-th loop frame, random() is a random selection function, and t is a random time point selected from the time series; represents a mask sequence; represents the predicted value;

[0031] Masker function It is expressed as:

[0032] (3)

[0033] in Indicates the sequence number of the loopback frame sequence.

[0034] In step 2, the predictor uses The spatiotemporal channel aggregator (SCA) stacks the learning cloud sequence to learn the dynamic trend of the cloud sequence;

[0035] The spatiotemporal channel aggregator SCA comprises a spatiotemporal aggregator SA and a channel aggregator CA;

[0036] The spatiotemporal aggregator SA comprises a feature decomposition module FD and a multi-order interaction gate MIG;

[0037] The spatiotemporal aggregator SA is formulated as:

[0038] (4)

[0039] Among them, Norm(X) is normalization, is the feature extraction function, is the transfer learning function, X is the input feature map;

[0040] The feature decomposition module FD is formulated as:

[0041]

[0042] (5)

[0043] in represents 1*1 convolution, represents global average pooling, Y is the output feature map, GELU is the activation function, and Z is the final output. represents element-by-element multiplication, β is a scaling factor with an initial value of zero, β∈ .

[0044] In step 2, the multi-stage interactive gate MIG includes two branches, namely, a gate branch and context branches ; In the context branch of the multi-order interaction gate MIG, more than two deep convolution DConvs are integrated to encode multi-order spatiotemporal features; Dynamic convolution DConvs of three sizes are used, with a dilation ratio of , capturing low, medium and high order interaction features in parallel; given the input features of the multi-order interaction gate MIG , after a 5×5 dynamic convolution with a dilation ratio of 1 After obtaining low-order interaction information, the output features are decomposed into three groups of outputs along the channel dimension: , and , ,in Indicates three different channel numbers, ; Enter the Encoding mid-level context information and high-level contextual information ;at last Concatenate to form multi-order context features ; In order to aggregate multi-order context features, SiLU activation function is added to the output of the gate branch and the context branch respectively, and the multi-order interaction gate MIG is formulated as:

[0045] (6).

[0046] In step 2, a channel aggregator CA is introduced at the output of the spatiotemporal aggregator SA to perform adaptive channel redistribution in the high-dimensional hidden space and eliminate information redundancy in the channel; the overall structure formula is:

[0047]

[0048] (7)

[0049] Channel aggregation function , using a branch to reduce channel projection, the input feature Convert to , to collect and redistribute channel information; Represents the transformed feature map; , ;

[0050] Channel Aggregation Functions The formula is:

[0051] (8)

[0052] Among them, α is the channel scaling factor with an initial value of zero.

[0053] The essence of cloud map prediction is space-time series prediction. The space-time series prediction algorithm based on deep learning can be summarized into three frameworks, namely single-input single-output framework (Single-In-Single-Out, SISO), multiple-input multiple-output framework (Multiple-In-Multiple-Out, MIMO) and multiple-input single-output framework (Multiple-In-Single-Out, MISO). According to the characteristics of cloud map motion, the present invention designs a cloud map prediction algorithm (IASCACP) based on implicit autoregressive space-time channel aggregation strategy by combining the characteristics of MIMO and SISO under the MISO framework.

[0054] Beneficial effects: In order to solve the problem of lack of spatiotemporal correlation in cloud images, the present invention introduces an implicit autoregressive codec, which stacks the predicted single-frame images into the input queue to participate in the prediction of the next frame, and gives the model autoregressive characteristics to capture the spatiotemporal correlation information in the sequence. At the same time, a mask module and a true mapping module are added in the encoding stage. Through learning, the error accumulation caused by future frame recursion is shielded, thereby increasing the robustness of the model. In addition, a spatiotemporal channel aggregator is introduced in the prediction module, wherein the spatiotemporal aggregator has the function of balancing the interaction of multi-order features, and can effectively aggregate low-order local features and medium- and high-order contextual aggregation features. The channel aggregator is used to adaptively redistribute channel features of the multi-order features output by the spatiotemporal aggregator to reduce channel feature redistribution, increase the effectiveness of multi-order features, and thereby increase prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is the IASCACP framework diagram.

[0056] Figure 2 It is a schematic diagram of implicit autoregressive encoder-decoder.

[0057] Figure 3 It is a schematic diagram of the spatiotemporal aggregator structure.

[0058] Figure 4 It is a schematic diagram of the channel aggregator structure.

[0059] Figure 5 It is a qualitative comparison diagram of cloud map predictions of multi-frame prediction models.

[0060] Figure 6 This is a schematic diagram of the frame-by-frame comparison of the predictions of the multi-frame prediction model MSE and SSIM.

[0061] Figure 7 This is a qualitative comparison diagram of the prediction of the multi-frame prediction model MovingMNIST. DETAILED DESCRIPTION

[0062] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0063] The multi-input single-output framework MISO combines the advantages of the single-input single-output framework SISO and the multi-input multiple-output framework MIMO. It uses multiple frames to predict a single frame, and stacks the single-frame predicted image to the input end to participate in the prediction of the next frame. This framework can solve the error accumulation problem existing in the SISO framework and the image correlation loss problem existing in the MIMO framework to a certain extent. In view of the cloud image motion characteristics and the MISO framework characteristics, this embodiment proposes a satellite cloud image prediction method based on the implicit autoregressive spatiotemporal channel aggregation strategy (Implicit autoregressivespatiotemporal channel aggregation algorithm for cloud image prediction, IASCACP) based on MISO as the basic framework. Specifically, it includes the following steps:

[0064] The autoregressive algorithm has temporal and spatial correlation but has error accumulation, while the non-autoregressive algorithm has no error accumulation but lacks temporal and spatial correlation. To address this problem, an implicit autoregressive codec is introduced, which stacks the predicted single-frame image into the input queue to participate in the prediction of the next frame, giving the model autoregressive characteristics. At the same time, a mask module is added in the encoding stage. Through learning, the error information caused by future frame recursion is shielded, thereby increasing the robustness of the model.

[0065] For spatiotemporal sequence prediction tasks such as cloud image motion prediction, which have high feature complexity and nonlinear motion trends, more attention should be paid to global spatiotemporal context information rather than local information during the prediction stage. To address this problem, a spatiotemporal channel aggregator (SCA) is introduced into the predictor, where the spatiotemporal aggregator (SA) has the function of balancing the interaction of multi-order features and can effectively aggregate low-order local features and medium- and high-order contextual aggregation features. The channel aggregator (CA) is used to adaptively redistribute the multi-order features output by the spatiotemporal aggregator to reduce channel feature redundancy, increase the effectiveness of multi-order features, and then increase prediction accuracy.

[0066] IASCACP network model:

[0067] The IASCACP network model adopts an implicit autoregressive strategy based on the MISO framework and has certain recursive characteristics. In order to weaken the error accumulation caused by recursion, the model adds a mask module after the encoder and a true mapping module (True Mapping, TM) at the prediction output. At the same time, a predictor is designed to selectively focus on the high-order contextual spatiotemporal features in the cloud sequence and aggregate them with low-order local features to comprehensively learn the spatiotemporal motion trends. The IASCACP framework is as follows Figure 1 shown.

[0068] The IASCACP network model adopts an encoder, predictor, and decoder structure.

[0069] The length of the sequence image at time t is n Input encoder, ,in , , and Respectively represent the frame length, number of channels, height and width of the sequence image; R represents all time series images;

[0070] Assume the loop frame sequence at time t is , then the predicted frame at time t is ;

[0071] The prediction process of the IASCACP network model is expressed as:

[0072] (1)

[0073] in , and denote the encoder, predictor and decoder respectively; is the position encoding function, Mask is the masker, and Concat is the stacking operation.

[0074] Combining the characteristics of MIMO and SISO framework, the decoder is an implicit autoregressive codec, and the implicit autoregressive codec structure is as follows: Figure 2 The decoder includes a positional encoding module PE (Positional Encoding, PE), a spatiotemporal refinement module STR and a mask module; the decoder also includes a prediction frame output port, and a real mapping module TM is introduced into the prediction frame output port;

[0075] like Figure 2It can be seen that the frame sequences in the strategy are divided into observation frames, loopback frames and masked frames. The observation frame sequence is the result obtained by the input sequence through the encoder. Its step size remains unchanged as time increases and always has the same encoding value. The loopback frame sequence is initially filled with zero tensors. As the time step advances, the predicted frames are gradually output and filled into the loopback frame sequence after passing through the TrueMapping module TM (TM). The loopback frame sequence is then passed through the masker Mask to obtain the masked frame sequence. The position encoding module PE is used to model the position relationship in the features and enhance the spatiotemporal correlation between sequences. The masked frame sequence and the observation frame sequence are stacked and combined with the data generated by the position encoding module PE, and then output to the predictor to learn the spatiotemporal trend changes, and then decoded by the decoder to obtain the current predicted frame.

[0076] The spatio-temporal refinement module (STR) refines the decoded frames. While calculating the loss of the predicted frame and the label frame, the real mapping module TM converts the predicted frame into a loop frame sequence that is more in line with the real features. The loop frame sequence is then masked and used together with the input frame to predict the next frame. In order to reduce the training cost during model training, the real future frame is filled into the loop frame sequence in the training phase, and the predicted frame is filled into the loop frame sequence in the verification phase.

[0077] The core of the implicit autoregressive codec is the mask module, which can randomly mask the redundant information in the loop frame sequence during the training phase, provide incomplete contextual spatiotemporal sequence information, and then increase the robustness of the model and reduce the impact of error stacking on the model. The operation process of the model training phase is: set the input frame to , the real future frame is , after passing through the implicit autoregressive encoder-decoder, we get:

[0078] (2)

[0079] in, Predict the frame length for the target, is the observation frame sequence, is the loop frame sequence, is the position encoder in the encoding module PE, is the i-th observation frame, is the i-th loop frame, random() is a random selection function, and t is a random time point selected from the time series; represents a mask sequence; represents the predicted value;

[0080] Masker function It is expressed as:

[0081] (3)

[0082] in Indicates the sequence number of the loopback frame sequence;

[0083] Although the random masking of the loop frame sequence by the mask module can enhance the robustness of the network, the feature vector directly extracted from the predicted frame cannot reflect the feature vector of the real future frame. In the model verification and testing phase, the prediction error may worsen as the prediction deepens. To solve this problem, a real mapping module TM is introduced at the predicted frame output port. This module can project the feature vector of the predicted frame to make it closer to the feature vector in the actual future frame. In the actual code, the structure of the real mapping module TM is the same as that of the encoding module.

[0084] Predictor:

[0085] The predictor uses The dynamic change trend of the learning cloud graph sequence is stacked by a spatiotemporal channel aggregator SCA. The spatiotemporal channel aggregator SCA includes a spatiotemporal aggregator SA and a channel aggregator CA. Traditional neural networks with local perception or context aggregation characteristics tend to interact in an extreme order, that is, only process low-order local information or high-order global information, while ignoring the most differentiated mid-order interaction information. In order to effectively capture the context information with multi-order balanced interactions, a spatiotemporal aggregator SA is introduced, whose structure is as follows Figure 3 shown.

[0086] GAP is Global Average Pooling (GAP), and Subtract means subtracting two values. The spatiotemporal aggregator SA includes a feature decomposition module (Feature Decomposition, FD) and a multi-order interaction gate (Multi-order Interaction Gate, MIG). As a pure convolutional structure, the spatiotemporal aggregator SA can extract multi-order features, so that the network has both static and adaptive local perception. Its formula is expressed as:

[0087] (4)

[0088] Among them, Norm(X) is normalization, is the feature extraction function, is the transfer learning function, X is the input feature map;

[0089] The feature decomposition module FD can dynamically remove trivial feature interactions, using convolution To obtain low-order interactions of common local textures, use global average pooling The high-order interactions covering the global complex shapes are obtained. The eigendecomposition module FD formulation is expressed as:

[0090]

[0091] (5)

[0092] Where Y is the output feature map, GELU is the activation function, and Z is the final output. represents element-by-element multiplication, β is a scaling factor with an initial value of zero, which is used to Reweight the complementary interaction component, β∈ .

[0093] In addition, the multi-stage interactive gate MIG includes two branches, namely the gate branch and context branches In the context branch of the multi-order interaction gate MIG, multiple deep convolution DCnvs are integrated to encode multi-order spatiotemporal features. Three sizes of dynamic convolution DCnv are used, with a dilation ratio of , capturing low, medium and high order interaction features in parallel. Given the multi-order interaction gate MIG input features , after a 5×5 dynamic convolution with a dilation ratio of 1 After obtaining low-order interaction information, the output features are decomposed into three groups of outputs along the channel dimension: , and , ,in Indicates three different channel numbers, ; Enter the Encoding mid-level context information and high-level contextual information .at last Concatenate to form multi-order context features In order to aggregate multi-order context features, SiLU activation function is added to the output of the gate branch and the context branch respectively. This function has the gating characteristics of Sigmoid function and is relatively stable in training. In summary, the multi-order interaction gate MIG is formulated as:

[0094] (6)

[0095] A channel aggregator CA is introduced at the output of the spatiotemporal aggregator SA to perform adaptive channel redistribution in the high-dimensional hidden space and eliminate information redundancy in the channel. The structure of the channel aggregator CA is as follows: Figure 4The overall structural formula is:

[0096]

[0097] (7)

[0098] Aggregate function for the channels , using a branch to reduce channel projection, the input feature Convert to , to collect and redistribute channel information; Represents the transformed feature map; , ;

[0099] Channel Aggregation Functions The formula is:

[0100] (8)

[0101] Among them, α is the channel scale factor with an initial value of zero, which is used to Redistribution of complementary channel interaction signals.

[0102] The movement of cloud maps is complex. The introduction of the spatiotemporal aggregator SA enables the prediction model to fully obtain the low-, medium-, and high-order feature interactions of the time series cloud map, and learn the spatiotemporal movement trend of the cloud map in multiple dimensions. The introduction of the CA module plays a filtering role in the multi-order spatiotemporal interaction information learned by the spatiotemporal aggregator SA, eliminates the redundant information in the channel, enhances the learning efficiency of the predictor, and improves the final prediction accuracy.

[0103] In a specific embodiment of the present invention, IASCACP is experimented on the FY-4A satellite cloud image dataset and the MovingMNIST dataset (the dataset contains labeled frames). The number of CL-CNN layers in the encoding and decoding module of IASCACP is set to 4, the number of SCA stacking layers in the decoder is 5, and the initial learning rate is 0.00001. All models use the L1+L2 loss function and the Adam optimizer. The experimental hardware conditions are NVIDIA GeForceRTX 3060 12G GPU, Intel Core i5-12400F CPU, and 32G memory.

[0104] FY-4A cloud image data set experiment: In the summer, the Yangtze River Delta region is subject to strong solar radiation, high ground temperature, unstable atmosphere, and easy to form convective clouds. As the temperature rises, the growth of updrafts in the troposphere is easy to form severe convective weather such as thunderstorms and tornadoes. Compared with other seasons, the cloud image in the Yangtze River Delta region in summer moves faster, and the instability and nonlinearity of the cloud image are more significant. Therefore, it is important to carry out prediction research on the cloud image over the Yangtze River Delta region in summer. FY-4A is China's second-generation geostationary meteorological satellite, which is equipped with 14 radiation imaging channels that can cover multiple bands such as visible light, near infrared, mid-infrared and far infrared. In order to obtain all-day observable images of the cloud image, the data of the long-wave infrared channel in the L1 level data of the FY-4A multi-channel imager is selected for the cloud image prediction experiment.

[0105] Cloud map samples of the Yangtze River Delta region from June to October in the summer of 2021 were selected with an interval of 1 hour. After filtering out bad and wrong images, 3488 cloud maps were obtained. According to the sliding value of the sequence length of 8, a total of 3481 cloud map sequences were obtained. 2780 sequences were randomly selected as training sets and 701 sequences were used as validation sets. The longitude and latitude range of the cloud map samples is from 115°2'E to 123°5'E east longitude and 26°6'N to 35°2'N north latitude. The pixel size is 256×256 and the spatial resolution is 4km. The STA-GRU Set to 4. After experimental evaluation, considering both practicality and effectiveness, the algorithm has the best comprehensive performance for 4h cloud map prediction. Cloud map prediction with a longer time span will cause the prediction accuracy to drop sharply, and the prediction results will not have practical value.

[0106] In addition, in order to comprehensively analyze the cloud map prediction results, the experiment uses mean square error (MSE), mean absolute error (MAE), structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) to evaluate the predicted image. SSIM is an indicator to measure the similarity between two pictures, ranging from -1 to 1. The closer the value is to 1, the more similar the two pictures are. Peak signal-to-noise ratio (PSNR) is an indicator to measure image quality, usually in dB. The higher the value, the better the quality of the predicted cloud map.

[0107] Analysis of ablation experiment results of cloud image dataset:

[0108] In order to evaluate the effectiveness of each module within IASCACP, IASCACP is split for ablation experiments, and MSE and SSIM are used as evaluation indicators. The experimental results are shown in Table 1.

[0109] Table 1

[0110] First, the MASK module is removed. At this time, the loop frame sequence will be directly stacked with the observation frame sequence without masking, and then sent to the predictor together with the output of PE. In this state, the error accumulation caused by the model recursion will be amplified, the robustness of the model will be weakened, and the prediction accuracy of the model will be significantly reduced. As a position encoding module, the PE module can enhance the position relationship between features and make up for the absolute position modeling ability of the CNN network. When the PE module is removed alone, the spatiotemporal features input to the predictor lack spatiotemporal correlation, which leads to a decrease in prediction quality. The STR and TM modules refine and realistically map the output of the network respectively. As shown in the table, when the two modules are removed, the prediction accuracy of the model is also attenuated to varying degrees. The results of the above ablation experiments prove the effectiveness and innovation of each module of the model.

[0111] Comparative experimental results of cloud image dataset: The MISO model IASCACP is compared with the new SISO models CrevNet and SmartCrevNet, and the new MIMO models SimVP and FACP on the FY-4A satellite cloud image dataset to explore the application of different prediction frameworks in the field of cloud image prediction. The quantitative evaluation is shown in Table 2.

[0112] Table 2

[0113] Table 2 shows the performance comparison of five different prediction algorithms under three frameworks on the FY-4A satellite dataset. From the perspective of MSE, MAE, and PSNR, the SISO model is better than the MIMO model overall, while the SSIM index of the MIMO model is better than the SISO model. This is because SSIM judges the quality of the image based on the brightness, contrast, and structure of the image. The error stacking of the SISO model will increase the noise information in the predicted image, change the brightness, texture and other information of the predicted image, and thus lead to a relatively low SSIM value. As a MISO prediction model, IASCACP combines the characteristics of the two frameworks. Compared with the SISO model, it reduces the impact of error stacking on the prediction. Compared with the MIMO model, it introduces recursive characteristics to enhance the spatiotemporal correlation of the predicted data. IASCACP is better than other framework algorithms in terms of MSE, MAE, and PSNR, and the SSIM index is on par with FACP. In summary, the IASCACP model of the MISO framework is better than the prediction models of the SISO and MIMO frameworks.

[0114] Qualitative comparison of IASCACP with CrevNet, SmartCrevNet, SimVP, and FACP models in cloud image prediction is shown below. Figure 5 As shown ( Figure 5Here, Input represents input, T represents time in hours, and T values ​​of 1 to 8 correspond to 1 to 8 hours, i.e. 8 time points, and GroundTruth represents the actual situation).

[0115] like Figure 5 It can be seen that both CrevNet and SmartCrevNet under the SISO framework accurately predicted the cloud trajectory. In the example, the prediction accuracy of SimVP dropped sharply in the second frame, and the subsequent movement trend was quite different from the actual trend. The existence of the dual-path feature extraction module of the FACP module enhanced the local and global dependencies in the sequence, and made targeted improvements to the problems existing in the MIMO framework. Compared with SimVP, the prediction accuracy of FACP decayed more slowly after the second frame. Overall, SmartCrevNet and IASCACP have better prediction effects in the example. The decoupling processing of different broken clouds in the last two frames of IASCACP prediction is more accurate than SmartCrevNet.

[0116] Frame-by-frame comparison of the predictions of the three framework models on the cloud map dataset Figure 6 As shown in the figure, it can be seen that SmartCrevNet and IASCACP have the smallest fluctuations under the MSE evaluation standard, while FACP and IASCACP have the slowest decay under the SSIM standard.

[0117] MovingMNIST dataset experiment: The MovingMNIST moving handwritten digit dataset is the most widely used dataset in spatiotemporal sequence prediction. This dataset can randomly generate numbers and make irregular movements in a 64×64 grid. The number of numbers generated by the dataset, the speed of movement, and the length of the generated sequence are all customizable. In theory, the dataset contains an infinite number of non-repeating sequences, which is very consistent with the experimental requirements. The experiment uses a randomly created dataset from MovingMNIST. The dataset is set to 10,000 sequences, of which the training set consists of 8,000 sequences and the validation set consists of 2,000 sequences. The sequence length is 10 and the batch size is set to 16.

[0118] Analysis of comparative experimental results on the MovingMNIST dataset: In order to further study the prediction performance of the prediction model, the five models under the three frameworks are compared on the MovingMNIST dataset. The quantitative evaluation is shown in Table 3.

[0119] Table 3

[0120] As shown in Table 3, different from the results of the cloud map dataset, SimVP and FACP under the MIMO framework are more accurate than CrevNet and SmartCrevNet under the SISO framework in terms of MSE and SSIM evaluation indicators. This is because MIMO-type prediction models are more suitable for small-scale spatiotemporal sequence prediction tasks with simpler features. As a MISO-type model that combines the advantages of the SISO and MISO frameworks, IASCACP has a more impressive performance on the MovingMNIST dataset, and both indicators are better than other models.

[0121] Figure 7 Three sets of prediction examples of the multi-frame prediction model under the MovingMNIST dataset are shown. The first set of "73" examples has a relatively simple motion trend, and there is no overlap when the numbers move. All five models have achieved accurate predictions. A closer look shows that the opening of the "3" in the last frame predicted by CrevNet, SmartCrevNet, and SimVP is sticky, while FACP and IASCACP are more accurate in predicting this area. The "36" example has been in a fusion state during the input stage, and the "3" in the label has some noise interference. Overall, in the "36" example, the five models automatically ignored the noise information in the "3" when predicting, and accurately predicted the motion trend. Among them, the "6" predicted by CrevNet has a large deformation, and the two numbers in the SmartCrevNet prediction result have more adhesion. The IASCACP prediction result is the most accurate, and the number state is the clearest. In the "22" example, the two numbers are in a sticky or overlapping state during the prediction stage, which puts forward further requirements for the performance of the prediction model. Specifically, in the "22" example, the five models accurately predicted the first two frames of numbers, and the two numbers after the third frame completely overlapped. The overlap of the last frame of numbers predicted by CrevNet and SmartCrevNet was fuzzy. SimVP and FACP learned wrong trend information during the prediction stage, resulting in anomalies at the top of the last two frames. Combining the above three groups of prediction examples, IASCACP's prediction results are the most accurate, and its ability to maintain the predicted target state is the best as the prediction progresses.

[0122] The present invention proposes a satellite cloud image prediction algorithm IASCACP based on the implicit autoregressive spatiotemporal channel aggregation strategy based on the MISO-based prediction framework. The algorithm combines the advantages of the two prediction frameworks of SISO and MIMO. Specifically, an implicit autoregressive codec is introduced to add autoregressive characteristics to the model to alleviate the problem of missing spatiotemporal correlation under the MIMO framework. A mask module and a true mapping module are added in the encoding stage and the decoding stage respectively to eliminate the error accumulation caused by the autoregressive characteristics. In addition, in order to predict the cloud image with unstable and nonlinear motion trends, a spatiotemporal channel aggregator SCA is introduced in the predictor, wherein the spatiotemporal aggregator SA is used to obtain the low-, medium- and high-order feature interaction information of the sequence cloud image, and the channel aggregator CA is used to adaptively redistribute the multi-order interaction information, thereby reducing the redundancy of channel features, improving the effectiveness of multi-order spatiotemporal information, and thus increasing the prediction accuracy. The IASCACP was experimentally compared with the cutting-edge models under the SISO and MIMO frameworks on the FY-4A and MovingMNIST datasets. The results showed that IASCACP outperformed other models in all indicators, proving that the model is more reliable and effective than the SISO and MIMO framework models in the field of cloud map prediction.

[0123] The present invention provides a satellite cloud image prediction method based on an implicit autoregressive spatiotemporal channel aggregation strategy. There are many methods and approaches to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. Satellite cloud image prediction method based on implicit autoregressive spatiotemporal channel aggregation strategy, It is characterized in that The following steps are involved: Step 1, establish an IASCACP network model based on implicit autoregressive spatiotemporal channel aggregation strategy; Step 2, executing an implicit autoregressive training strategy on the IASCACP network model to obtain a trained IASCACP network model; Step 3: Use the trained IASCACP network model to predict satellite cloud images.

2. The method according to claim 1, It is characterized in that In step 1, the IASCACP network model includes an encoder, a predictor and a decoder; The length of the sequence image at time t is n Input encoder, ,in , , and Respectively represent the frame length, number of channels, height and width of the sequence image; R is all time series images; Assume the loop frame sequence at time t is , then the predicted frame at time t is ; The prediction process of the IASCACP network model is expressed as: （1） in , and denote the encoder, predictor and decoder respectively; is the position encoding function, Mask is the masker, and Concat is the stacking operation.

3. The method according to claim 2, It is characterized in that In step 1, the decoder is an implicit autoregressive codec, including a position encoding module PE, a spatiotemporal refinement module STR and a mask module; the decoder also includes a prediction frame output port, and a real mapping module TM is introduced at the prediction frame output port, and the real mapping module TM is used to project a feature vector on the prediction frame.

4. The method according to claim 3, It is characterized in that In step 1, the frame sequence processed in the IASCACP network model includes an observation frame sequence, a loop frame sequence and a mask frame sequence; The observation frame sequence is the result obtained by passing the input sequence through the encoder, and the step length of the observation frame sequence remains unchanged as time increases, and always has the same encoding value; The loop frame sequence is initially filled with zero tensors. As the time step advances, the predicted frames are gradually output and filled into the loop frame sequence after passing through the real mapping module TM. The loop frame sequence is then passed through a masker Mask to obtain a mask frame sequence; The mask frame sequence and the observation frame sequence are stacked to obtain a stacked sequence, and the position encoding module PE is used to model the position relationship in the stacked sequence and enhance the spatiotemporal correlation between sequences; The mask frame sequence and the observation frame sequence are stacked and combined with the data generated by the position encoding module PE, and then output to the predictor to learn the spatiotemporal trend changes, and then decoded by the decoder to obtain the current predicted frame.

5. The method according to claim 4, It is characterized in that In step 1, the spatiotemporal refinement module STR refines the decoded frame; while calculating the loss of the predicted frame and the label frame, the predicted frame is converted into a loop frame sequence through the real mapping module TM, and the loop frame sequence is then masked and used together with the input frame to predict the next frame; When training the model, the real future frames are filled into the loop frame sequence in the training phase, and the predicted frames are filled into the loop frame sequence in the verification phase.

6. The method according to claim 5, It is characterized in that In step 1, the mask module is used to randomly mask redundant information in the loop frame sequence during the training phase to provide incomplete contextual spatiotemporal sequence information.

7. The method according to claim 6, It is characterized in that Step 2 includes: setting the input frame to , the real future frame is , after passing through the implicit autoregressive encoder-decoder, we get: （2） in, Predict the frame length for the target, is the observation frame sequence, is the loop frame sequence, is the position encoder in the encoding module PE, is the i-th observation frame, is the i-th loop frame, random() is a random selection function, and t is a random time point selected from the time series; represents a mask sequence; represents the predicted value; Masker function It is expressed as: (3) in Indicates the sequence number of the loopback frame sequence.

8. The method according to claim 7, It is characterized in that In step 2, the predictor uses The spatiotemporal channel aggregator (SCA) stacks the learning cloud sequence to learn the dynamic trend of the cloud sequence; The spatiotemporal channel aggregator SCA comprises a spatiotemporal aggregator SA and a channel aggregator CA; The spatiotemporal aggregator SA comprises a feature decomposition module FD and a multi-order interaction gate MIG; The spatiotemporal aggregator SA is formulated as: (4) Among them, Norm(X) is normalization, is the feature extraction function, is the transfer learning function, X is the input feature map; The feature decomposition module FD is formulated as: (5) in represents 1*1 convolution, represents global average pooling, Y is the output feature map, GELU is the activation function, and Z is the final output. represents element-by-element multiplication, β is a scaling factor with an initial value of zero, β∈ .

9. The method according to claim 8, It is characterized in that In step 2, the multi-stage interactive gate MIG includes two branches, namely, a gate branch and context branches ; More than two deep convolutions DConv are integrated in the context branch of the multi-order interaction gate MIG to encode multi-order spatiotemporal features; Using three sizes of dynamic convolution DConv, expansion ratio , capturing low-, medium-, and high-order interaction features in parallel; Given the input features of the multi-order interactive gate MIG , after a 5×5 dynamic convolution with a dilation ratio of 1 After obtaining low-order interaction information, the output features are decomposed into three groups of outputs along the channel dimension: , and , ,in Indicates three different channel numbers, ; Enter the Encoding mid-level context information and high-level contextual information ;at last Concatenate to form multi-order context features ; In order to aggregate multi-order context features, SiLU activation function is added to the output of the gate branch and the context branch respectively, and the multi-order interaction gate MIG is formulated as: (6)。 10. The method according to claim 9, It is characterized in that In step 2, a channel aggregator CA is introduced at the output of the spatiotemporal aggregator SA to perform adaptive channel redistribution in the high-dimensional latent space and eliminate information redundancy in the channel; The overall structural formula is: (7) Channel aggregation function , using a branch to reduce channel projection, the input feature Convert to , to collect and redistribute channel information; Represents the transformed feature map; , ; Channel Aggregation Functions The formula is: (8) Among them, α is the channel scaling factor with an initial value of zero.

Images