Satellite image sequence prediction method and device based on deep spatio-temporal coding model

Abstract

The application discloses a satellite image sequence prediction method and device based on a deep space-time coding and decoding model, and the method comprises the following steps: processing Fengyun satellite data to obtain a satellite image sequence, and constructing a corresponding training set, a verification set and a test set, wherein the satellite image sequence comprises a first satellite image sequence; secondly, inputting the training set into a deep learning model to train the deep learning model; inputting the verification set into the trained deep learning model to adjust parameters of the trained deep learning model; inputting the test set into the deep learning model with the adjusted parameters, and outputting a predicted second satellite image sequence; calculating errors between the second satellite image sequence and the first satellite image sequence, and performing secondary training on the deep learning model based on the errors. Through the method and the device, the motion field of the historical satellite image sequence can be accurately modeled, and a long-time, stable and accurate satellite image sequence prediction service can be provided.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and remote sensing image data processing technology, specifically to a satellite image sequence prediction method and apparatus based on a deep spatiotemporal encoding and decoding model. Background Technology

[0002] Satellite image sequence extrapolation aims to accurately predict future image sequences using historical satellite images. However, due to the extreme complexity and uncertainty of atmospheric motion, achieving long-term and accurate forecasting of satellite image sequences remains a significant challenge. Currently, methods for satellite image sequence extrapolation can be broadly categorized into two types: 1. Satellite image sequence extrapolation methods based on traditional methods; 2. Satellite image sequence extrapolation methods based on deep learning.

[0003] Traditional satellite image sequence extrapolation methods extrapolate satellite image sequences by estimating the motion field of historical satellite image sequences. These methods are simple and easy to use, but their shortcomings are also very obvious. The estimation of the motion field is very coarse and cannot accurately model complex motion patterns in the atmosphere, resulting in low accuracy of the extrapolated satellite images.

[0004] Deep learning-based satellite image sequence extrapolation methods directly utilize neural networks to model complex spatiotemporal patterns in satellite image sequences, thereby generating future satellite images. Compared to traditional methods, deep learning-based satellite image sequence extrapolation methods predict satellite image sequences more accurately. However, there are still some limitations when using existing technologies to simulate the motion field of historical satellite image sequences, and the predicted satellite images often lack some texture details.

[0005] In summary, there are two bottleneck problems in using existing technologies for satellite image sequence prediction:

[0006] First, it is not possible to make effective long-term predictions. At present, the most advanced methods can only extrapolate satellite image sequences with a duration of 2 hours. At the same time, as the prediction duration increases, the objects in the later predicted images will become smaller and smaller, which will seriously affect the quality of the extrapolated images.

[0007] Second, the image texture details generated by the existing deep learning extrapolation method are not clear enough and cannot meet the application needs in real-world scenarios. Summary of the Invention

[0008] To address two problems existing in the prior art, this application proposes a satellite image sequence prediction method and apparatus based on a deep spatiotemporal encoding and decoding model, so as to provide a long-term, stable and accurate satellite image sequence prediction service.

[0009] To achieve the above objectives, according to one aspect of this application, a satellite image sequence prediction method based on a deep spatiotemporal encoding and decoding model is proposed, the method comprising:

[0010] The data from the Fengyun satellite is processed to obtain satellite image sequences, and corresponding training, validation, and test sets are constructed. The satellite image sequences include a first satellite image sequence.

[0011] The training set is input into the deep learning model to train the deep learning model.

[0012] The validation set is input into the trained deep learning model, and the parameters of the trained deep learning model are adjusted.

[0013] The test set is input into the parameter-adjusted deep learning model, which then outputs the predicted second satellite image sequence.

[0014] The error between the second satellite image sequence and the first satellite image sequence is calculated, and the deep learning model is trained a second time based on the error.

[0015] Optionally, the deep learning model includes a spatial encoder, a spatiotemporal feature extractor, and a spatial decoder. Training the deep learning model includes:

[0016] The spatial encoder is used to extract spatial feature maps from the satellite image sequence;

[0017] The spatiotemporal features of the spatial feature map are determined using the spatiotemporal feature extractor.

[0018] The spatiotemporal features are decoded using the spatial decoder.

[0019] Optionally, the spatial encoder is used to extract spatial feature maps from the satellite image sequence, including: extracting the spatial feature maps based on multiple serial deformable spatial coding modules.

[0020] Optionally, each deformable spatial coding module includes two serial residual modules, and the spatial feature map is extracted based on multiple serial deformable spatial coding modules, including:

[0021] The residual module is used to perform standard convolution, deformable convolution, activation, and residual calculation operations on the satellite image sequence in sequence.

[0022] Optionally, a deformable convolution operation is performed on the satellite image sequence, including: performing deformable convolution calculation based on Formula 1 and Formula 2.

[0023] Formula 1:

[0024] Formula 2:

[0025] in, The offset is represented by p0, a position on the feature map is represented by p0, w(.) represents the weight matrix, G(.) represents the bilinear interpolation operation, and x(.) represents the feature map.

[0026] Optionally, the spatiotemporal features of the spatial feature map are determined using the spatiotemporal feature extractor, including:

[0027] The spatiotemporal feature extractor includes multiple stacked spatiotemporal coding units.

[0028] The spatiotemporal features are extracted using the spatiotemporal coding unit.

[0029] Optionally, the spatiotemporal features are extracted using the spatiotemporal coding unit, including:

[0030] The spatiotemporal coding unit includes a temporal attention module, a spatial attention module, and a spatiotemporal fusion module.

[0031] The corresponding time features are extracted using the time attention module.

[0032] The spatial attention module is used to extract the corresponding spatial features;

[0033] The time features and spatial features are fused using the spatiotemporal fusion module.

[0034] Optionally, the corresponding temporal features are extracted using the temporal attention module, including:

[0035] Based on Formula 3, the corresponding time features are extracted. Formula 3:

[0036] Among them, U g Indicates gating, T represents the short-term characteristics of the (k-1)th layer. long This indicates the fusion feature.

[0037] Optionally, the spatial attention module is used to extract corresponding spatial features, including:

[0038] Based on Formula 4, the corresponding spatial features are extracted. Formula 4:

[0039] Among them, U r Indicates gating, S represents the spatial characteristics of the (k-1)th layer. long This indicates the characteristics of the integrated space.

[0040] Optionally, the spatiotemporal fusion module is used to fuse the temporal features and the spatial features, including:

[0041] Using gated U ts and U st Based on Formulas 5 and 6, the time features of the next moment are generated. Spatial features

[0042] Formula 5:

[0043] Formula Six:

[0044] Among them, T final S represents the time characteristic. final Represents spatial characteristics.

[0045] Optionally, the spatial decoder is used to decode the spatiotemporal features, including:

[0046] The spatial decoder comprises two decoding layers, each layer consisting of two deconvolutional modules and one activation function.

[0047] The spatiotemporal features are decoded based on Formulas 7 and 8.

[0048] Formula 7: h′ i =ReLu(Deconv(Deconv(h) i ))),

[0049] Formula 8: h″ i =ReLu(Deconv(Deconv(h′) i ))),

[0050] Among them, h i This represents the spatiotemporal characteristics of the input.

[0051] Optionally, training the deep learning model includes:

[0052] The deep learning model is trained using a joint function of MSE and MAE.

[0053] Optionally, the deep learning model is trained using a joint function of MSE and MAE, including:

[0054] The deep learning model is trained based on formulas nine, ten, and eleven.

[0055] Formula 9: L(G) = L MSE +λ1L MAE +λ2L GAN (G),

[0056] Formula 10:

[0057] Formula 11:

[0058] Where L(G) represents the loss function of the generator G, L MSE L represents the Mean Square Error (MSE) loss function. MAE L represents the Mean Absolute Error (MAE) loss function. GAN (G) denotes the adversarial loss function in the generator G. Represents the i-th predicted image The predicted label in the discriminator, D(x) t+i ) represents the i-th real image x t+i In the discriminator, the predicted label is L(D), which represents the loss function of the discriminator D.

[0059] According to another aspect of this application, a satellite image sequence prediction device based on a deep spatiotemporal encoding and decoding model is proposed, the device comprising:

[0060] The processing module is used to process Fengyun satellite data to obtain satellite image sequences and construct corresponding training sets, validation sets and test sets, wherein the satellite image sequences include a first satellite image sequence.

[0061] The training module is used to input the training set into the deep learning model and train the deep learning model.

[0062] The adjustment module is used to input the validation set into the trained deep learning model and adjust the parameters of the trained deep learning model.

[0063] The testing module is used to input the test set into the parameter-adjusted deep learning model and output the predicted second satellite image sequence.

[0064] The secondary training module is used to calculate the error between the second satellite image sequence and the first satellite image sequence, and to perform secondary training on the deep learning model based on the error.

[0065] Optionally, the training module is used for:

[0066] The deep learning model includes a spatial encoder, a spatiotemporal feature extractor, and a spatial decoder.

[0067] The spatial encoder is used to extract spatial feature maps from the satellite image sequence;

[0068] The spatiotemporal features of the spatial feature map are determined using the spatiotemporal feature extractor.

[0069] The spatiotemporal features are decoded using the spatial decoder.

[0070] Optionally, the training module is used to: extract the spatial feature map based on multiple serial deformable spatial coding modules.

[0071] Optionally, each deformable space coding module includes two serial residual modules, the training module being used for:

[0072] The residual module is used to perform standard convolution, deformable convolution, activation, and residual calculation operations on the satellite image sequence in sequence.

[0073] Optionally, the training module is used for:

[0074] Deformable convolution is calculated based on Formula 1 and Formula 2.

[0075] Formula 1:

[0076] Formula 2:

[0077] in, The offset is represented by p0, a position on the feature map is represented by p0, w(.) represents the weight matrix, G(.) represents the bilinear interpolation operation, and x(.) represents the feature map.

[0078] Optionally, the training module is used for:

[0079] The spatiotemporal feature extractor includes multiple stacked spatiotemporal coding units.

[0080] The spatiotemporal features are extracted using the spatiotemporal coding unit.

[0081] Optionally, the training module is used for:

[0082] The spatiotemporal coding unit includes a temporal attention module, a spatial attention module, and a spatiotemporal fusion module.

[0083] The corresponding time features are extracted using the time attention module.

[0084] The spatial attention module is used to extract the corresponding spatial features;

[0085] The time features and spatial features are fused using the spatiotemporal fusion module.

[0086] Optionally, the training module is used for:

[0087] Based on Formula 3, the corresponding time features are extracted. Formula 3:

[0088] Among them, U g Indicates gating, T represents the short-term characteristics of the (k-1)th layer. long This indicates the fusion feature.

[0089] Optionally, the training module is used for:

[0090] Based on Formula 4, the corresponding spatial features are extracted. Formula 4:

[0091] Among them, U r Indicates gating, S represents the spatial characteristics of the (k-1)th layer. long This indicates the characteristics of the integrated space.

[0092] Optionally, the training module is used for:

[0093] Using gated U ts and U st Based on Formulas 5 and 6, the time features of the next moment are generated. Spatial features

[0094] Formula 5:

[0095] Formula Six:

[0096] Among them, T final S represents the time characteristic. final Represents spatial characteristics.

[0097] Optionally, the training module is used for:

[0098] The spatial decoder comprises two decoding layers, each layer consisting of two deconvolutional modules and one activation function.

[0099] The spatiotemporal features are decoded based on Formulas 7 and 8.

[0100] Formula 7: h′ i =ReLu(Deconv(Deconv(h) i ))),

[0101] Formula 8: h″ i =ReLu(Deconv(Deconv(h′) i ))),

[0102] Among them, h iThis represents the spatiotemporal characteristics of the input.

[0103] Optionally, the training module is used for:

[0104] The deep learning model is trained using a joint function of MSE and MAE.

[0105] Optionally, the training module is used for:

[0106] The deep learning model is trained based on formulas nine, ten, and eleven.

[0107] Formula 9: L(G) = L MSE +λ1L MAE +λ2L GAN (G),

[0108] Formula 10:

[0109] Formula 11:

[0110] Where L(G) represents the loss function of the generator G, L MSE L represents the Mean Square Error (MSE) loss function. MAE L represents the Mean Absolute Error (MAE) loss function. GAN (G) denotes the adversarial loss function in the generator G. Represents the i-th predicted image The predicted label in the discriminator, D(x) t+i ) represents the i-th real image x t+i In the discriminator, the predicted label is L(D), which represents the loss function of the discriminator D.

[0111] Based on the above technical solution, the satellite image sequence prediction method and device based on the deep spatiotemporal encoding and decoding model have at least the following beneficial effects:

[0112] 1. This invention utilizes a deep learning model to simulate complex spatiotemporal features. The model is trained, its parameters are adjusted, and it is tested and evaluated using training, validation, and test sets. Based on the acquired spatiotemporal features, the model is predicted to predict future satellite image sequences. Finally, the model is retrained based on the error between the predicted satellite image sequence and the actual sequence. This model can accurately model the motion field of historical satellite image sequences, thereby effectively improving the accuracy of satellite image sequence prediction.

[0113] 2. This invention establishes a spatiotemporal feature extractor in a deep learning model to determine the spatiotemporal features of the spatial feature map. At the same time, it adds special gating to optimize the spatiotemporal memory unit, so that the model can capture high-level and low-level spatial information, as well as long-term and short-term temporal information without limitation. Through this method and device, high-quality satellite image extrapolation can be achieved, providing a stable and accurate 4-hour satellite image sequence forecast service.

[0114] 3. This invention uses an adversarial generative function to train a deep learning model. The generator G (i.e., the deep learning model) continuously optimizes its own parameters, making the predicted sequence closer and closer to the real sequence. In the end, the satellite image output by the deep learning model has higher quality. Attached Figure Description

[0115] The accompanying drawings, which form part of this specification, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0116] Figure 1 This is a flowchart of a satellite image sequence prediction method based on a deep spatiotemporal encoding and decoding model according to an embodiment of this application;

[0117] Figure 2 This is a model structure diagram of the satellite image sequence prediction based on the spatiotemporal encoding and decoding model of deep learning in this application;

[0118] Figure 3 These are the training steps for the deep learning model in this application;

[0119] Figure 4 This is a structural diagram of the spatial encoder model in the deep learning model of this application;

[0120] Figure 5 This is a structural diagram of the spatiotemporal feature extractor and spatiotemporal coding unit in the deep learning model of this application;

[0121] Figure 6 This is a flowchart of the computation of the spatiotemporal coding unit in the deep learning model of this application;

[0122] Figure 7 This is a diagram of the spatial decoder model structure in the deep learning model of this application;

[0123] Figure 8 This is a schematic diagram of a satellite image sequence prediction device based on a deep spatiotemporal encoding and decoding model according to an embodiment of this application. Detailed Implementation

[0124] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0125] The present invention will be further described in detail below with reference to specific embodiments, which should not be construed as limiting the scope of protection claimed by the present invention.

[0126] Example

[0127] According to one aspect of this application, a satellite image sequence prediction method based on a deep spatiotemporal encoding and decoding model is proposed.

[0128] exist Figure 1 The diagram shows a flowchart of a satellite image sequence prediction method based on a deep spatiotemporal encoding and decoding model, according to an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0129] S1 processes Fengyun satellite data to obtain satellite image sequences and constructs corresponding training, validation, and test sets.

[0130] The acquired satellite image sequence includes the first satellite image sequence, which is the actual satellite image sequence. Prior to this, the original Fengyun satellite data needs to undergo relevant processing operations.

[0131] In one specific embodiment, the obtained raw Fengyun satellite data is in HDF format. The data processing operation required after obtaining the raw data is to convert the raw Fengyun satellite data format to JPG or PNG format.

[0132] Assuming the raw satellite data obtained in the above specific embodiment contains 14 channels, corresponding to 14 different wavebands, the data processing operation needs to convert the corresponding channels into JPG or PNG format images according to actual requirements. Taking the 12th channel, the long-wave infrared channel, as an example, the data value range of the long-wave infrared channel is [180, 320]. The lower the value, the stronger the feedback. According to the requirements, the data of this channel is converted into a grayscale index in the range of [0, 255], so as to reduce the brightness temperature of this channel and increase the grayscale value. The specific conversion formula is as follows:

[0133]

[0134] Where x represents the measured value of the long-wave infrared channel.

[0135] After the above data processing operations, the processed Fengyun satellite data and acquired satellite image sequences are used to construct corresponding training sets, validation sets, and test sets according to a certain ratio.

[0136] S2, Input the training set into the deep learning model and train the deep learning model.

[0137] exist Figure 2 The diagram shows a structure of a deep learning-based satellite image sequence prediction model according to an embodiment of this application. Figure 2 As shown, the deep learning model includes a spatial encoder, a spatiotemporal feature extractor, and a spatial decoder.

[0138] The training process of the deep learning model can be simply summarized as follows: First, the spatial encoder extracts spatial feature maps from the input satellite image sequence; second, the spatiotemporal feature extractor captures the spatiotemporal dynamic relationship of the above spatial features; and finally, the spatial decoder decodes these features into pixel-level predicted images.

[0139] For example, in one specific embodiment, suppose X = {x1, x2, x3, ..., x...} t}, x i ∈m×n represents satellite images from frame t of history, Y={x t+1 x t+2 x t+3 , ..., x t+k Let} represent k future satellite images, where m and n represent the length and width of the satellite image, respectively. This model aims to generate a sequence of k satellite images. And it strives to approximate the actual satellite image sequence Y as closely as possible. Its workflow is as follows:

[0140]

[0141]

[0142]

[0143] Where Encoder(.) represents the spatial encoder, STExtractor(.) represents the spatiotemporal feature extractor, and Decoder(.) represents the spatial decoder. ST represents the feature map extracted by the spatial encoder, and ST represents the spatiotemporal features captured by the spatiotemporal feature extractor.

[0144] like Figure 3 As shown, in a specific embodiment of this application, the training steps of the deep learning model are as follows:

[0145] S201, the spatial encoder is used to extract spatial feature maps from the satellite image sequence.

[0146] The spatial encoder extracts the spatial feature map based on multiple serial deformable spatial coding modules. This module can obtain a larger spatial receptive field with fewer parameters, which is beneficial for capturing the spatial features of satellite images and modeling the relationship between frames.

[0147] In another specific embodiment of this application, such as Figure 4 As shown, the spatial encoder consists of four deformable spatial encoding modules connected in series. Satellite images from different times share the deformable spatial encoder to extract spatial features. The first and third encoding modules have downsampling capabilities, which can reduce image size and decrease the computational load of the model.

[0148] Specifically, each deformable spatial coding module includes two serial residual modules. The residual modules are used to perform standard convolution, deformable convolution, activation, and residual calculation operations on the satellite image sequence in sequence to extract spatial features.

[0149] For example, for the current time i, satellite image X i After passing through the spatial encoder, the corresponding spatial features can be obtained. Where f1(.) represents the first layer of deformable space module, f2(.) represents the second layer of deformable space module, and so on. The following will take the first layer of deformable space coding module as an example to introduce the calculation process of the residual module.

[0150] Assuming Block(.) represents the residual module, the detailed calculation process of the residual module is as follows:

[0151] x i =Conv(X i )

[0152] x i =DeformableConv(x i )

[0153] x i =ReLu(x i )

[0154] x i =x i +X i

[0155] Where Conv(.) represents the standard convolution operation, DeformableConv(.) represents the deformable convolution operation, and ReLu(.) represents the activation function.

[0156] In other words, through the above calculations, the standard convolution operation, deformable convolution operation, activation operation, and residual calculation operation of the first layer residual module were completed, resulting in x.i =Block(X) i After passing through two layers of residual modules, the final spatial feature x can be obtained. i =Block(Block(X) i )).

[0157] In particular, deformable convolution differs from standard convolution. During deformable convolution, an offset is required for each location in the feature map, and estimating this offset is complex and inefficient. Therefore, this application improves the deformable convolution operation by incorporating bilinear interpolation to enhance the computational efficiency of the offset during deformable convolution.

[0158] Specifically, this application performs deformable convolution calculations based on Formula 1 and Formula 2.

[0159] Formula 1:

[0160] Formula 2:

[0161] in, Let p0 represent a position on the feature map, q represent the four nearest pixels to p0, corresponding to the four directions: up, down, left, and right, w(.) represent the weight matrix, G(.) represent the bilinear interpolation operation, and x(.) represent the feature map.

[0162] Based on Equations 1 and 2, the calculation of deformable convolution involves two processes. First, an offset is introduced into traditional convolution using Equation 1. Second, the offset pixel is obtained through bilinear interpolation in Equation 2, thus realizing the calculation of deformable convolution.

[0163] S202, the spatiotemporal features of the spatial feature map are determined using the spatiotemporal feature extractor.

[0164] To accurately simulate complex spatiotemporal patterns in satellite image sequences, this application proposes using a spatiotemporal feature extractor to determine the spatiotemporal features of the spatial feature map. In one embodiment of this application, such as... Figure 5 The overall structure is shown in the figure. The spatiotemporal feature extractor includes multiple stacked spatiotemporal coding units. The spatiotemporal features are extracted using the spatiotemporal coding units. The spatiotemporal coding units at different layers are used to extract spatiotemporal features at different levels.

[0165] In one specific embodiment of this application, the spatiotemporal coding unit includes a temporal attention module, a spatial attention module, and a spatiotemporal fusion module, such as... Figure 6 The diagram shows the calculation flowchart of the spatiotemporal coding unit in this embodiment.

[0166] S2021, The corresponding time features are extracted using the time attention module.

[0167] Specifically, the time attention module is used to extract the corresponding time features, including the correlation between the motion features at the current moment and the motion features at historical moments.

[0168] The calculation process of the time attention module can be simply summarized as follows: First, utilize the features of the previous time step. Characteristics of historical moments Calculate the attention score, and then perform a weighted summation of the obtained attention scores to obtain the time feature T containing historical information. att The detailed calculation process is as follows:

[0169]

[0170]

[0171]

[0172]

[0173] Where * represents convolution, ⊙ represents Hadamard product, and SUM(.) represents summation. The above calculation process can be understood as the temporal characteristics at time t-1 of the k-th layer. After convolution operation It is transformed into T′, and then T′ is compared with the historical moment. Perform the Hadamard product operation and sum the results to obtain the similarity score p. i The attention score α is obtained through the softmax function operation. j Finally, a weighted summation is performed to obtain the time feature T. att .

[0174] The time feature T obtained through the above calculations att It contains long-term time information, but in addition to long-term time information, this module also needs to extract short-term time information. Therefore, this application designs a gating U in the time attention module. t ,

[0175]

[0176] Where σ(.) represents the sigmoid activation function, which can be understood as the gated U t The temporal features of the k-th layer at time t-1 First, perform a convolution operation W tf It is then calculated using the sigmoid activation function.

[0177] Secondly, through the gated Ut To integrate long-term time information T att With short-term information The calculation process is as follows:

[0178]

[0179] Specifically, the fusion feature T obtained through the above formula long It only contains time information from the k-th layer, lacking time information from the (k-1)-th layer. To more accurately capture time features, this application also designs a gated U in the time attention module. g ,

[0180] U g =σ(W gf *T long )

[0181] This can be understood as the gated U g For fusion feature T long First, perform a convolution operation W gf It is then calculated using the sigmoid activation function.

[0182] Secondly, using gated U g T long Short-term characteristics of the (k-1)th layer Further integration is performed, and the corresponding time features are extracted based on Formula 3.

[0183] Formula 3:

[0184] in, T represents the short-term characteristics of the (k-1)th layer. long This indicates the fusion feature.

[0185] S2022, The spatial attention module is used to extract the corresponding spatial features.

[0186] Specifically, the spatial attention module is used to extract corresponding spatial features, including capturing the correlation between the appearance features at the current moment and the appearance features at historical moments.

[0187] The computation process of the spatial attention module can be summarized as follows: First, calculate the spatial features of the previous time step. Spatial features of the previous floors The attention scores are then weighted and summed to obtain the spatial features S with multiple layers of information. att The detailed calculation process is as follows:

[0188]

[0189]

[0190]

[0191]

[0192] The above calculation process can be understood as first analyzing the spatial features of the k-th layer at time t-1. After convolution operation Transform it into S′, and then combine S′ with the spatial features of the i-th layer above. The similarity score q is obtained by performing the Hadamard product operation. i Then, the attention score β is calculated using the softmax function. j The final attention score β will be obtained j Compared with the spatial features of the i-th layer above The spatial feature S is obtained by performing a weighted summation. att .

[0193] The spatial feature S obtained through the above calculations att This module contains multi-layered spatial information, especially high-level information. Similar to the temporal attention module, to ensure the accuracy of the results, in addition to the high-level spatial information, it is also necessary to acquire low-level spatial information. Therefore, this application designs a gating U in this module. s To integrate high-level spatial information S att With low-level spatial information The calculation process is as follows:

[0194]

[0195]

[0196] In addition, due to the spatial characteristics of the previous layer This module has the greatest impact on the spatial characteristics of the current level, therefore it integrates higher-level spatial information S. att With low-level spatial information Based on this, it also incorporates the spatial features of the (k-1)th layer. This makes the spatial features acquired at the current level more accurate.

[0197] The specific implementation method is as follows: An additional gating U is designed within the spatial attention module. r ,

[0198] U r =σ(W rf *S long )

[0199] This can be understood as the gated U r To integrate spatial features S longFirst, perform a convolution operation W rf It is then calculated using the sigmoid activation function.

[0200] Secondly, using gated U r S long Spatial characteristics of the (k-1)th layer Further integration is achieved by extracting the corresponding spatial features based on Formula 4.

[0201] Formula 4:

[0202] in, S represents the spatial characteristics of the (k-1)th layer. long This indicates the characteristics of the integrated space.

[0203] S2023, The time features and the spatial features are fused using the spatiotemporal fusion module.

[0204] The temporal features T generated by the temporal attention module and the spatial attention module were calculated through the above steps S2021 and S2022. final Spatial characteristics S final In this step, the spatiotemporal fusion module will be used to fuse the aforementioned temporal and spatial features.

[0205] Because temporal and spatial characteristics are mutually restrictive, this application designs two gated U-channels in the spatiotemporal fusion module. ts and U st ,

[0206] U ts =σ(W rts *T final )

[0207] U st =σ(W rst *S final )

[0208] This can be understood as the gated U ts For time feature T final First, perform a convolution operation W rts Then, the gated U is calculated using the sigmoid activation function. st For spatial features S final First, perform a convolution operation W rst It is then calculated using the sigmoid activation function.

[0209] Secondly, using gated U ts and U st Based on Formulas 5 and 6, the time features of the next moment are generated. Spatial features

[0210] Formula 5:

[0211] Formula Six:

[0212] Where ⊙ represents the Hadama product, T final S represents the time characteristic. final Represents spatial characteristics.

[0213] S203, the spatiotemporal features are decoded using the spatial decoder.

[0214] The spatial decoder corresponds to the spatial encoder mentioned in S201. The spatial decoder is responsible for decoding high-dimensional spatiotemporal features into pixel-level predicted images.

[0215] In one specific embodiment of this application, such as Figure 7 As shown, the spatial decoder consists of four spatial decoding modules in series. The first and third decoding modules have upsampling functions to ensure that the size of the predicted image is consistent with the size of the real image. Satellite image predictions at different times can share the same spatial decoding module.

[0216] Specifically, each spatial decoder includes two serial decoding components, and each decoding component includes two standard deconvolution layers and a ReLU activation function layer.

[0217] Taking time t+i as an example, the spatial decoder is used to extract the spatiotemporal features h generated by the spatiotemporal feature extractor in S202. t+i Decoding into predicted image Where g1(.) represents the first-layer spatial decoding module, g2(.) represents the second-layer spatial decoding module, and so on. Taking the first-layer spatial decoding module g1(.) as an example, the spatiotemporal features can be decoded using formulas seven and eight:

[0218] Formula 7: h′ i =ReLu(Deconv(Deconv(h) i )))

[0219] Formula 8: h″ i =ReLu(Deconv(Deconv(h′) i )))

[0220] Among them, h i Let h represent the spatiotemporal features of the input, Deconv(.) represent the standard deconvolution operation, and ReLU(.) represent the activation function. That is, through the above calculations, the spatiotemporal features h of the input are transformed. iTwo standard deconvolution operations were performed to obtain Deconv(Deconv(h)). i Finally, the spatiotemporal features h″ of the decoded data are obtained through an activation function. i .

[0221] S3, input the validation set into the trained deep learning model and adjust the parameters of the trained deep learning model.

[0222] After the deep learning model is trained in step S2, the validation set constructed in S1 is input into the trained model. Based on the validation results, the parameters in the model are adjusted appropriately to improve the model performance.

[0223] S4. Input the test set into the parameter-adjusted deep learning model and output the predicted second satellite image sequence.

[0224] After verifying the model's performance and adjusting the parameters in step S3, the test set constructed in S1 is input into the adjusted deep learning model, which outputs the predicted second satellite sequence, i.e., the future satellite image sequence.

[0225] S5, calculate the error between the second satellite image sequence and the first satellite image sequence, and perform secondary training on the deep learning model based on the error.

[0226] The predicted second satellite sequence in S4 is compared with the first satellite image sequence to calculate the prediction error of the deep learning model. Specifically, in this application, the joint function of MSE and MAE is used to evaluate the prediction effect of the model, and then the deep learning model is trained a second time.

[0227] To improve the visual effect of the predicted image, this application also introduces an adversarial generative function to train the deep learning model. The adversarial generative function includes a generator G and a discriminator D. The generator G is used to generate samples, and the discriminator D judges the authenticity of the current sample by learning the difference between the real sample and the sample generated by the generator G.

[0228] Specifically, in this application, the trained deep learning prediction model is used as the generator G, and a three-layer convolutional stacked structure is used as the discriminator D.

[0229] The training of the entire model is divided into two stages: First, the discriminator D is trained to judge the authenticity of the current sample, that is, to distinguish whether the current sample is a real satellite image sequence or a satellite image sequence predicted by the generator G; Second, the sample generated by the generator G is used to deceive the discriminator. During this process, the generator G will continuously optimize its own parameters so that the discriminator D will judge its predicted satellite image sequence as a real satellite image sequence.

[0230] This process can be understood as making the predicted sequence increasingly closer to the real sequence. The generator's generated results are more accurate, resulting in higher quality images output by the final deep learning model. Training ends when the discriminator D reaches an equilibrium point, meaning it can no longer distinguish between real and fake samples generated by the generator G.

[0231] Specifically, based on the loss function L of the deep learning model, the deep learning model is trained using Equations 9, 10, and 11:

[0232] Formula 9: L(G) = L MSE +λ1L MAE +λ2L GAN (G),

[0233] Formula 10:

[0234] Formula 11:

[0235] Where L(G) represents the loss function of the generator G, L MSE L represents the Mean Square Error (MSE) loss function. MAE L represents the Mean Absolute Error (MAE) loss function. GAN (G) denotes the adversarial loss function in the generator G. Represents the i-th predicted image The predicted label in the discriminator, D(X) t+i ) represents the i-th real image x t+i In the discriminator, the predicted label is L(D), which represents the loss function of the discriminator D.

[0236] During the training process of the aforementioned deep learning model: First, the parameters of the generator G are frozen, and the i-th real image x is obtained. t+i Predicted label D(x) in the discriminator t+i ), the i-th predicted image Predicted label in discriminator D Train the discriminator D according to Formula 11.

[0237] Secondly, the parameters of the frozen discriminator D are calculated, and the mean square error L is determined. MSE Mean absolute error L MAE Obtain the i-th predicted image. Predicted label in discriminator D The generator G is trained according to Formulas 9 and 10.

[0238] The model training is terminated when the quality of the images generated by the model no longer improves, that is, when the loss function L(G) of the generator G and the loss function L(D) of the discriminator D tend to converge and reach equilibrium.

[0239] Specifically, when training the model, this application sets the parameters λ1 and λ2 to 0.1 and 0.001 respectively, uses the Adam optimizer to train the prediction model, sets the learning rate to 1e-4, and sets the hidden state dimension of the spatiotemporal extractor to 128.

[0240] According to another aspect of this application, a satellite image sequence prediction device based on a deep spatiotemporal encoding and decoding model is proposed, the device comprising:

[0241] exist Figure 8 The diagram shows a schematic of a satellite image sequence prediction device based on a deep spatiotemporal encoding and decoding model according to an embodiment of this application. Figure 8 As shown, the device includes a processing module 81, a training module 82, an adjustment module 83, a testing module 84, and a secondary training module 85.

[0242] The processing module 81 is used to process Fengyun satellite data to obtain satellite image sequences and construct corresponding training sets, validation sets and test sets, wherein the satellite image sequences include a first satellite image sequence.

[0243] The training module 82 is used to input the training set into the deep learning model and train the deep learning model.

[0244] The adjustment module 83 is used to input the validation set into the trained deep learning model and adjust the parameters of the trained deep learning model.

[0245] The test module 84 is used to input the test set into the parameter-adjusted deep learning model and output the predicted second satellite image sequence.

[0246] The secondary training module 85 is used to calculate the error between the second satellite image sequence and the first satellite image sequence, and to perform secondary training on the deep learning model based on the error.

[0247] The training module 82 is specifically used for:

[0248] The deep learning model includes a spatial encoder, a spatiotemporal feature extractor, and a spatial decoder.

[0249] The spatial encoder is used to extract spatial feature maps from the satellite image sequence.

[0250] The spatiotemporal features of the spatial feature map are determined using the spatiotemporal feature extractor.

[0251] The spatiotemporal features are decoded using the spatial decoder.

[0252] The training module 82 is specifically used to: extract the spatial feature map based on multiple serial deformable spatial coding modules.

[0253] Specifically, each deformable space coding module includes two serial residual modules, and the training module is used for:

[0254] The residual module is used to perform standard convolution, deformable convolution, activation, and residual calculation operations on the satellite image sequence in sequence.

[0255] The training module 82 is specifically used for:

[0256] Deformable convolution is calculated based on Formula 1 and Formula 2.

[0257] Formula 1:

[0258] Formula 2:

[0259] in, The offset is represented by p0, a position on the feature map is represented by p0, w(.) represents the weight matrix, G(.) represents the bilinear interpolation operation, and x(.) represents the feature map.

[0260] The training module 82 is specifically used for:

[0261] The spatiotemporal feature extractor comprises multiple stacked spatiotemporal coding units.

[0262] The spatiotemporal features are extracted using the spatiotemporal coding unit.

[0263] The training module 82 is specifically used for:

[0264] The spatiotemporal coding unit includes a temporal attention module, a spatial attention module, and a spatiotemporal fusion module.

[0265] The corresponding time features are extracted using the time attention module.

[0266] The spatial attention module is used to extract the corresponding spatial features.

[0267] The time features and spatial features are fused using the spatiotemporal fusion module.

[0268] The training module 82 is specifically used for: extracting the corresponding time features based on Formula 3.

[0269] Formula 3: Among them, U g Indicates gating, T represents the short-term characteristics of the (k-1)th layer. long This indicates the fusion feature.

[0270] The training module 82 is specifically used for: extracting the corresponding spatial features based on formula four.

[0271] Formula 4: Among them, U r Indicates gating, S represents the spatial characteristics of the (k-1)th layer. long This indicates the characteristics of the integrated space.

[0272] The training module 82 is specifically used for: utilizing gating U ts and U st Based on Formulas 5 and 6, the time features of the next moment are generated. Spatial features

[0273] Formula 5:

[0274] Formula Six:

[0275] Among them, T final S represents the time characteristic. final Represents spatial characteristics.

[0276] The training module 82 is specifically used for:

[0277] The spatial decoder comprises two decoding layers, each layer consisting of two deconvolutional modules and one activation function.

[0278] The spatiotemporal features are decoded based on Formulas 7 and 8.

[0279] Formula 7: h′ i =ReLu(Deconv(Deconv(h) i ))),

[0280] Formula 8: h″ i =ReLu(Deconv(Deconv(h′) i ))),

[0281] Among them, h i This represents the spatiotemporal characteristics of the input.

[0282] The training module 82 is used for:

[0283] The deep learning model is trained using a joint function of MSE and MAE.

[0284] The training module 82 is used for:

[0285] The deep learning model is trained based on formulas nine, ten, and eleven.

[0286] Formula 9: L(G) = L MSE +λ1L MAE +λ2L GAN (G),

[0287] Formula 10:

[0288] Formula 11:

[0289] Where L(G) represents the loss function of the generator G, L MSE L represents the Mean Square Error (MSE) loss function. MAE L represents the Mean Absolute Error (MAE) loss function. GAN (G) denotes the adversarial loss function in the generator G. Represents the i-th predicted image The predicted label in the discriminator, D(x) t+i ) represents the i-th real image x t+i In the discriminator, the predicted label is L(D), which represents the loss function of the discriminator D.

[0290] It should be understood that the satellite image sequence prediction device based on the deep spatiotemporal coding and decoding model is consistent with the embodiment description of its corresponding satellite image sequence prediction method based on the deep spatiotemporal coding and decoding model, so it will not be repeated in this embodiment.

[0291] In summary, as can be seen from the above description, the above embodiments of the satellite image sequence prediction method and apparatus based on the deep spatiotemporal encoding and decoding model achieve the following technical effects:

[0292] 1. This invention utilizes a deep learning model to simulate complex spatiotemporal features. The model is trained, its parameters are adjusted, and it is tested and evaluated using training, validation, and test sets. Based on the acquired spatiotemporal features, the model is predicted to predict future satellite image sequences. Finally, the model is retrained based on the error between the predicted satellite image sequence and the actual sequence. This model can accurately model the motion field of historical satellite image sequences, thereby effectively improving the accuracy of satellite image sequence prediction.

[0293] 2. This invention establishes a spatiotemporal feature extractor in a deep learning model to determine the spatiotemporal features of the spatial feature map. At the same time, it adds special gating to optimize the spatiotemporal memory unit, so that the model can capture high-level and low-level spatial information, as well as long-term and short-term temporal information without limitation. Through this method and device, high-quality satellite image extrapolation can be achieved, providing a stable and accurate 4-hour satellite image sequence forecast service.

[0294] 3. This invention uses an adversarial generative function to train a deep learning model. The generator G (i.e., the deep learning model) continuously optimizes its own parameters, making the predicted sequence closer and closer to the real sequence. In the end, the satellite image output by the deep learning model has higher quality.

[0295] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0296] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0297] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0298] It should be noted that, in the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0299] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A satellite image sequence prediction method based on a deep spatiotemporal encoding and decoding model, characterized in that, include: Process Fengyun satellite data to obtain satellite image sequences, and construct corresponding training sets, validation sets, and test sets, wherein the satellite image sequences include a first satellite image sequence; The training set is input into the deep learning model to train the deep learning model; The deep learning model includes a spatial encoder, a spatiotemporal feature extractor, and a spatial decoder. Training the deep learning model includes: The spatial encoder is used to extract spatial feature maps from the satellite image sequence; Determining the spatiotemporal features of the spatial feature map using the spatiotemporal feature extractor includes: The spatiotemporal feature extractor includes multiple stacked spatiotemporal coding units. The spatiotemporal features are extracted using the spatiotemporal coding unit, which includes a temporal attention module, a spatial attention module, and a spatiotemporal fusion module. The corresponding time features are extracted using the time attention module. The spatial attention module is used to extract the corresponding spatial features; The spatiotemporal fusion module is used to fuse the temporal features and the spatial features; The spatiotemporal features are decoded using the spatial decoder; The validation set is input into the trained deep learning model, and the parameters of the trained deep learning model are adjusted. The test set is input into the parameter-adjusted deep learning model, and the predicted second satellite image sequence is output. The error between the second satellite image sequence and the first satellite image sequence is calculated, and the deep learning model is trained a second time based on the error.

2. The method according to claim 1, characterized in that, Extracting spatial feature maps from the satellite image sequence using the space encoder includes: The spatial feature map is extracted based on multiple serial deformable spatial coding modules.

3. The method according to claim 2, characterized in that, Each deformable spatial coding module includes two serial residual modules. The spatial feature map is extracted based on multiple serial deformable spatial coding modules, including: The residual module is used to perform standard convolution, deformable convolution, activation, and residual calculation operations on the satellite image sequence in sequence.

4. The method according to claim 3, characterized in that, Performing deformable convolution operations on the satellite image sequence includes: Deformable convolution is calculated based on Formula 1 and Formula 2. Formula 1: ; Formula 2: ; in, Indicates the offset. This represents a location on the feature map. Represents the weight matrix. This represents the bilinear interpolation operation. Represents the feature map.

5. The method according to claim 1, characterized in that, Extracting corresponding temporal features using the aforementioned time attention module includes: Based on Formula 3, the corresponding time features are extracted. Formula 3: ,in, Indicates gating, This represents the short-term characteristics of the (k-1)th layer. This indicates the fusion feature.

6. The method according to claim 1, characterized in that, Extracting corresponding spatial features using the spatial attention module includes: Based on Formula 4, the corresponding spatial features are extracted. Formula 4: ,in, Indicates gating, Represents the spatial characteristics of the (k-1)th layer. This indicates the characteristics of the integrated space.

7. The method according to claim 1, characterized in that, The spatiotemporal fusion module is used to fuse the temporal features and the spatial features, including: Using gating and Based on Formulas 5 and 6, the time features of the next moment are generated. Spatial features , Formula 5: , Formula Six: , in, Indicates time characteristics, Represents spatial characteristics.

8. The method according to claim 1, characterized in that, Decoding the spatiotemporal features using the spatial decoder includes: The spatial decoder comprises two decoding layers, each layer consisting of two deconvolutional modules and one activation function. The spatiotemporal features are decoded based on Formulas 7 and 8. Formula 7: , Formula 8: , in, This represents the spatiotemporal characteristics of the input.

9. The method according to claim 1, characterized in that, Training the deep learning model includes: The deep learning model is trained using a joint function of MSE and MAE.

10. The method according to claim 9, characterized in that, The deep learning model is trained using a joint function of MSE and MAE, including: The deep learning model is trained based on formulas nine, ten, and eleven. Formula Nine: , Formula 10: , Formula 11: L , Where L(G) represents the loss function of the generator G, This represents the Mean Square Error (MSE) loss function. This represents the Mean Absolute Error (MAE) loss function. This represents the adversarial loss function in the generator G. Represents the i-th predicted image Predicted label in the discriminator Represents the i-th real image In the discriminator, the predicted label is L(D), which represents the loss function of the discriminator D.

11. A satellite image sequence prediction device based on a deep spatiotemporal encoding and decoding model, characterized in that, include: The processing module is used to process Fengyun satellite data to obtain satellite image sequences and construct corresponding training sets, validation sets and test sets, wherein the satellite image sequences include a first satellite image sequence; A training module is used to input the training set into a deep learning model and train the deep learning model. The deep learning model includes a spatial encoder, a spatiotemporal feature extractor, and a spatial decoder. The training module is used to extract spatial feature maps from the satellite image sequence using the spatial encoder; determine the spatiotemporal features of the spatial feature maps using the spatiotemporal feature extractor; and decode the spatiotemporal features using the spatial decoder. The spatiotemporal feature extractor includes multiple stacked spatiotemporal coding units. The spatiotemporal features are extracted using the spatiotemporal coding unit; The spatiotemporal coding unit includes a temporal attention module, a spatial attention module, and a spatiotemporal fusion module. The corresponding time features are extracted using the time attention module. The spatial attention module is used to extract the corresponding spatial features; The spatiotemporal fusion module is used to fuse the temporal features and the spatial features; The adjustment module is used to input the validation set into the trained deep learning model and adjust the parameters of the trained deep learning model. The testing module is used to input the test set into the parameter-adjusted deep learning model and output the predicted second satellite image sequence; The secondary training module is used to calculate the error between the second satellite image sequence and the first satellite image sequence, and to perform secondary training on the deep learning model based on the error.

12. The apparatus according to claim 11, characterized in that, The training module is used for: The spatial feature map is extracted based on multiple serial deformable spatial coding modules.

13. The apparatus according to claim 12, characterized in that, Each deformable spatial coding module includes two serial residual modules, the training module being used for: The residual module is used to perform standard convolution, deformable convolution, activation, and residual calculation operations on the satellite image sequence in sequence.

14. The apparatus according to claim 13, characterized in that, The training module is used for: Deformable convolution is calculated based on Formula 1 and Formula 2. Formula 1: , Formula 2: , in, Indicates the offset. This represents a location on the feature map. Represents the weight matrix. This represents the bilinear interpolation operation. Represents the feature map.

15. The apparatus according to claim 11, characterized in that, The training module is used for: Based on Formula 3, the corresponding time features are extracted. Formula 3: ,in, Indicates gating, This represents the short-term characteristics of the (k-1)th layer. This indicates the fusion feature.

16. The apparatus according to claim 11, characterized in that, The training module is used for: Based on Formula 4, the corresponding spatial features are extracted. Formula 4: ,in, Indicates gating, Represents the spatial characteristics of the (k-1)th layer. This indicates the characteristics of the integrated space.

17. The apparatus according to claim 11, characterized in that, The training module is used for: Using gating and Based on Formulas 5 and 6, the time features of the next moment are generated. Spatial features , Formula 5: , Formula Six: , in, Indicates time characteristics, Represents spatial characteristics.

18. The apparatus according to claim 11, characterized in that, The training module is used for: The spatial decoder comprises two decoding layers, each layer consisting of two deconvolutional modules and one activation function. The spatiotemporal features are decoded based on Formulas 7 and 8. Formula 7: , Formula 8: , in, This represents the spatiotemporal characteristics of the input.

19. The apparatus according to claim 11, characterized in that, The training module is used for: The deep learning model is trained using a joint function of MSE and MAE.

20. The apparatus according to claim 19, characterized in that, The training module is used for: The deep learning model is trained based on formulas nine, ten, and eleven. Formula Nine: , Formula 10: , Formula 11: L , Where L(G) represents the loss function of the generator G, This represents the Mean Square Error (MSE) loss function. This represents the Mean Absolute Error (MAE) loss function. This represents the adversarial loss function in the generator G. Represents the i-th predicted image Predicted label in the discriminator Represents the i-th real image In the discriminator, the predicted label is L(D), which represents the loss function of the discriminator D.

Images