A spatio-temporal fusion method for remote sensing images based on a generative adversarial network

Abstract

The application relates to the technical field of remote sensing, in particular to a remote sensing image space-time fusion method and device based on a generative adversarial network, which comprises the following steps: adopting a switchable multi-scale convolution attention network to respectively extract spatial features of different scales of a reference remote sensing image F1 at a t1 moment and time features of different scales of a predicted remote sensing image C2 at a t2 moment, obtaining a group of spatial features and a group of time features; performing feature fusion on the spatial features and the time features through a double-parallel attention feature fusion module, obtaining intermediate features used for image reconstruction; and inputting the intermediate features into an image reconstruction network to obtain a space-time fusion remote sensing image at the t2 moment, so that the method reduces the dependence on the number of reference images and improves the efficiency and accuracy of remote sensing image space-time fusion.

Description

Technical Field

[0001] This application relates to the field of remote sensing technology, and in particular to a method and apparatus for spatiotemporal fusion of remote sensing images based on generative adversarial networks. Background Technology

[0002] High spatiotemporal resolution remote sensing imagery plays a crucial role in monitoring natural disasters, vegetation cover, land use, and air pollution. However, due to technological and cost limitations, such images are often difficult to obtain. Satellites with high spatial resolution typically have long revisit periods. Conversely, satellites with short revisit periods usually provide lower spatial resolution. Spatiotemporal fusion technology offers a cost-effective method for acquiring high spatiotemporal resolution remote sensing imagery. This technology combines medium-resolution images from MODIS sensors with high-resolution images from Landsat satellites to generate a fused image with high spatial and temporal resolution.

[0003] Although existing spatiotemporal fusion methods can achieve good fusion results, current mainstream methods mostly rely on four to five high-quality reference remote sensing images as input, placing high demands on both image quality and quantity. Taking patent CN115731141A as an example, it proposes an innovative spatiotemporal fusion method for dynamic monitoring of moving targets. This method, based on a trained spatiotemporal fusion network model, needs to integrate multiple low-resolution and high-resolution images from different data sources and spanning multiple time points to accurately predict the high-resolution image at a specific moment. However, in practice, due to environmental factors, collecting a sufficient number of high-quality reference images is not easy, which directly limits the method's wide applicability and real-time performance. Therefore, reducing the model's requirement for the number of reference images is of great significance. Summary of the Invention

[0004] Therefore, it is necessary to provide a spatiotemporal fusion method and apparatus for remote sensing images based on generative adversarial networks that can reduce the number of reference images required by the model, in order to address the above-mentioned technical problems.

[0005] In a first aspect, this application provides a spatiotemporal fusion method for remote sensing images based on generative adversarial networks (GANs). The generator in the GAN includes a switchable multi-scale convolutional attention network and a dual parallel attention feature fusion module. The method includes:

[0006] Step S1: Obtain the reference remote sensing image F1 at time t1 and the predicted remote sensing image C2 at time t2; the reference remote sensing image at time t1 is a remote sensing image with low temporal resolution and high spatial resolution, and the predicted remote sensing image at time t2 is a remote sensing image with high temporal resolution and low spatial resolution.

[0007] Step S2: A switchable multi-scale convolutional attention network is used to extract spatial features at different scales of the reference remote sensing image F1 at time t1 and temporal features at different scales of the predicted remote sensing image C2 at time t2, resulting in a set of spatial features. and a set of time features

[0008] Step S3, spatial features and time characteristics Feature fusion is performed using a dual parallel attention feature fusion module to obtain intermediate features for image reconstruction. The dual-parallel attention feature fusion module includes a first attention module and a second attention module. The first attention module is a parallel fusion of global attention and local attention, while the second attention module is a parallel fusion of channel attention and spatial attention. The first attention module is based on spatial features... and time characteristics The initial fusion features are obtained, and the second attention module is based on the spatial features. Time characteristics Intermediate features are obtained from the initial fusion features.

[0009] Step S4, intermediate features The image is input into an image reconstruction network to obtain a spatiotemporal fused remote sensing image at time t2.

[0010] In one embodiment, the switchable multi-scale convolutional attention network includes at least one downsampled multi-scale convolutional attention block. Each downsampled multi-scale convolutional attention block includes a switchable normalization layer, a multi-scale convolutional attention layer, and a feedforward network layer. The multi-scale convolutional attention layer includes depthwise convolution, multi-branch depthwise strip convolution, and 1×1 convolution. The kernel size of each branch in the multi-branch depthwise strip convolution is set to 5, 7, and 11, respectively.

[0011] In one embodiment, local attention includes two 1×1 convolutional layers, and global attention includes a global average pooling layer and two 1×1 convolutional layers.

[0012] In one embodiment, spatial attention includes two 1×1 convolutional layers and two dilated convolutional layers.

[0013] In one embodiment, the porosity of the dilated convolutional layer is 4.

[0014] In one embodiment, the channel attention includes an average pooling branch and a max pooling branch, which are connected in parallel. The average pooling branch includes an average pooling layer and a first fully connected layer, and the max pooling branch includes a max pooling layer and a second fully connected layer.

[0015] In one embodiment, the remote sensing image spatiotemporal fusion method based on generative adversarial networks further includes:

[0016] Step S51: Combine the spatiotemporal fused remote sensing images at time t2. The input is fed into the discriminator for identification, and the identification result is obtained;

[0017] Step S52: Based on the identification results, the reference remote sensing image F1 at time t1, the predicted remote sensing image C2 at time t2, and the spatiotemporally fused remote sensing image at time t2. Calculate the value of the loss function;

[0018] Step S53: Update the parameters of the switchable multi-scale convolutional attention network, the dual parallel attention feature fusion module, and the discriminator according to the value of the loss function;

[0019] Step S54: Repeat steps S51 to S53 until the value of the loss function meets the preset conditions, and use the parameters of the current switchable multi-scale convolutional attention network, dual parallel attention feature fusion module and discriminator as the final determined model parameters.

[0020] In one embodiment, the loss function includes a conditional generation network total loss function and a discriminator loss function;

[0021] Conditional Generator Network Total Loss Function The expression is:

[0022]

[0023] In the formula, This represents the loss function of a least-squares generative adversarial network. Describes the L1 loss function. Represents the spectral loss function. Let represent the structural loss function, α represent the weights of the least-squares generative adversarial network loss function, β represent the weights of the L1 loss function, γ represent the weights of the spectral loss function, and δ represent the weights of the structural loss function; where,

[0024] Least squares generative adversarial network loss function The expression is:

[0025]

[0026] In the formula, F1 represents the reference remote sensing image at time t1, C2 represents the predicted remote sensing image at time t2, E represents the expectation, and G(F1, C2) represents the spatiotemporal fused remote sensing image at time t2 obtained by inputting the reference remote sensing image F1 at time t1 and the predicted remote sensing image C2 at time t2 into the generator. D(G(F1, C2), C2) represents the combination of the predicted remote sensing image C2 at time t2 and the spatiotemporally fused remote sensing image at time t2. The identification result obtained after inputting into the discriminator;

[0027] L1 loss function The expression is as follows:

[0028]

[0029] In the formula, K represents the number of pixels in the remote sensing image, T represents the real image matrix, P represents the predicted image matrix, and ||.||1 represents the L1 norm;

[0030] Spectral loss function The expression is as follows:

[0031]

[0032] In the formula, I is a tensor consisting of all 1s, and ||.||2 represents the L2 norm;

[0033] Structural loss function The expression is as follows:

[0034]

[0035] In the formula, l m V represents the brightness comparison at the j-th scale. N Let represent the weighting coefficients for brightness comparison at the j-th scale, n represent the number of highest-level scales, and c represent the weighting coefficients. j ρ represents the contrast comparison at the j-th scale. j s represents the weighting coefficient for contrast comparison at the j-th scale. j μ represents the structural comparison at the j-th scale. j Represents the weighting coefficients for structural comparisons at the j-th scale;

[0036] Discriminator loss function The expression is:

[0037]

[0038] In the formula, F2 represents the true high spatial resolution remote sensing image at time t2, and D(F2,C2) represents the discrimination result obtained after inputting the predicted remote sensing image C2 at time t2 and the true high spatial resolution remote sensing image F2 at time t2 into the discriminator.

[0039] In one embodiment, the image reconstruction network includes an upsampling residual module, which upsamples intermediate features through transposed convolutions and skip connections to obtain the spatiotemporally fused remote sensing image at time t2.

[0040] Secondly, this application provides a spatiotemporal fusion device for remote sensing images based on generative adversarial networks, the device comprising:

[0041] The remote sensing image acquisition module is used to acquire the reference remote sensing image F1 at time t1 and the predicted remote sensing image C2 at time t2; the reference remote sensing image at time t1 is a remote sensing image with low temporal resolution and high spatial resolution, and the predicted remote sensing image at time t2 is a remote sensing image with high temporal resolution and low spatial resolution.

[0042] The image feature extraction module is used to extract spatial features at different scales of the reference remote sensing image F1 at time t1 and temporal features at different scales of the predicted remote sensing image C2 at time t2 using a switchable multi-scale convolutional attention network, thereby obtaining a set of spatial features. and a set of time features

[0043] Image feature fusion module, used to integrate spatial features and time characteristics Feature fusion is performed using a dual parallel attention feature fusion module to obtain intermediate features for image reconstruction. The dual-parallel attention feature fusion module includes a first attention module and a second attention module. The first attention module is a parallel fusion of global attention and local attention, while the second attention module is a parallel fusion of channel attention and spatial attention. The first attention module is based on spatial features... and time characteristics The initial fusion features are obtained, and the second attention module is based on the spatial features. Time characteristics Intermediate features are obtained from the initial fusion features.

[0044] The spatiotemporal fusion remote sensing image generation module will generate intermediate features. The image is input into an image reconstruction network to obtain a spatiotemporal fused remote sensing image at time t2.

[0045] The aforementioned spatiotemporal fusion method and apparatus for remote sensing images based on generative adversarial networks (GANs) introduces a switchable multi-scale convolutional attention network and a dual parallel attention feature fusion module into the generator of the GAN. The switchable multi-scale convolutional attention network can adaptively adjust feature extraction at different scales, enabling the model to effectively capture key information even with limited reference images, significantly reducing the requirement for the number of reference images. Meanwhile, the dual parallel attention feature fusion module includes a first attention module that fuses global and local attention in parallel and a second attention module that fuses channel attention and spatial attention in parallel. The first attention module generates preliminary fusion features by capturing global trends and local details in the image, enhancing the richness and accuracy of feature representation. The second attention module further utilizes the spatial and temporal features of the remote sensing image, as well as the preliminary fusion features, and leverages the parallel enhancement of channel and spatial attention to deeply explore the intrinsic connections and importance differences between features, thereby generating more refined and discriminative intermediate features. Therefore, this method reduces the dependence on the number of reference images and improves the efficiency and accuracy of spatiotemporal fusion of remote sensing images by optimizing the image feature extraction process using a switchable multi-scale convolutional attention network and optimizing the feature fusion process using a dual parallel attention feature fusion module. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating a remote sensing image spatiotemporal fusion method based on generative adversarial networks in one embodiment.

[0047] Figure 2 This is a schematic diagram of the structure of a downsampling multi-scale convolutional attention block in one embodiment;

[0048] Figure 3 This is a schematic diagram of the structure of a dual-parallel attention feature fusion module in one embodiment;

[0049] Figure 4 This is a schematic diagram of the structure of a generative adversarial network in one embodiment;

[0050] Figure 5 This is a schematic diagram illustrating the detection results of different models on the CIA dataset in one embodiment.

[0051] Figure 6 This is a schematic diagram illustrating the detection results of different models on the LGC dataset in one embodiment.

[0052] Figure 7 This is a heatmap of the detection results of different models on the CIA dataset in one embodiment.

[0053] Figure 8 This is a heat map of the detection results of different models on the LGC dataset in one embodiment;

[0054] Figure 9 This is a structural block diagram of a remote sensing image spatiotemporal fusion device based on generative adversarial networks in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] Since the introduction of Generative Adversarial Networks (GANs), many GAN-based image processing models have emerged, playing a crucial role in image modeling and processing. The core architecture of GANs includes a generator and a discriminator: the generator learns from random noise to generate samples highly similar to real data, while the discriminator distinguishes whether the data comes from the real dataset or is a product of the generator. These two parts mutually reinforce each other through iterative training, achieving high-fidelity sample generation and high-precision discrimination. This application's remote sensing image spatiotemporal fusion method is based on an improved GAN for spatiotemporal fusion of remote sensing images. The generator in the improved GAN includes a Switchable Multi-scale Convolutional Attention Network (SMSCAN) and a Dual Parallel Attention Feature Fusion (DPAFF) module.

[0057] In one embodiment, such as Figure 1 As shown, a spatiotemporal fusion method for remote sensing images based on generative adversarial networks is provided. This embodiment illustrates the method by applying it to a server; however, it is understood that the method can also be applied to a terminal, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0058] Step S1: Obtain the reference remote sensing image F1 at time t1 and the predicted remote sensing image C2 at time t2.

[0059] The reference remote sensing image at time t1 is a remote sensing image with low temporal resolution and high spatial resolution, while the predicted remote sensing image at time t2 is a remote sensing image with high temporal resolution and low spatial resolution.

[0060] Step S2: A switchable multi-scale convolutional attention network is used to extract spatial features at different scales of the reference remote sensing image F1 at time t1 and temporal features at different scales of the predicted remote sensing image C2 at time t2, resulting in a set of spatial features. and a set of time features

[0061] The switchable multi-scale convolutional attention network comprises two branches: one branch extracts spatial features at different scales from the reference remote sensing image F1 at time t1, and the other branch extracts temporal features at different scales from the predicted remote sensing image C2 at time t2. Each branch includes multiple consecutive downsampled multi-scale convolutional attention blocks for extracting multi-scale spatiotemporal features from the image. Spatial features Time characteristics Represents the multi-scale spatial features of the i-th layer. Let N represent the multi-scale temporal features of the i-th layer, and N be the number of layers from which the multi-level features are extracted.

[0062] Specifically, each downsampling multi-scale convolutional attention block performs convolution operations using kernels of different scales to extract features at different scales and capture contextual information of varying ranges. This enables the network to identify and process features at different scales, from local details to global structure in an image. During feature extraction, the downsampling multi-scale convolutional attention block strengthens key features through weighted fusion while relatively weakening redundant or unimportant information. Furthermore, the downsampling multi-scale convolutional attention block can analyze the importance of each scale feature based on its relevance and importance, dynamically adjusting the weights of each scale feature according to the analysis results, so that more significant and valuable features receive more attention. This adaptive capability allows the network to automatically focus on key information in the image.

[0063] Meanwhile, a switchable normalization layer (SN) is added to the downsampling multi-scale convolutional attention block to replace the traditional BatchNorm. Switchable normalization is a highly flexible and data-driven adaptive normalization strategy. Its core advantage lies in its ability to learn and dynamically adjust key normalization parameters in the network to adapt to the characteristics of different tasks and datasets. Specifically, the switchable normalization layer optimizes the normalization process by simultaneously considering statistical information (mean and variance) from three dimensions: channel, layer, and mini-batch. This multi-dimensional consideration enables the switchable normalization layer to maintain excellent performance stability at various batch sizes, exhibiting strong robustness even in mini-batch scenarios. The design of the switchable normalization layer eliminates the dependence on sensitive hyperparameters, simplifies the network configuration process, and maintains the lightweight nature of the overall architecture. This not only reduces the implementation difficulty but also promotes its rapid deployment and efficient application in various application scenarios.

[0064] Step S3, spatial features and time characteristics Feature fusion is performed using a dual parallel attention feature fusion module to obtain intermediate features for image reconstruction.

[0065] Traditional attention mechanisms, while guiding the network to assign different feature weights based on feature importance, cannot fully integrate high- and low-resolution multi-scale features obtained during feature extraction. Remote sensing images typically have six channels with significant spatial variations; therefore, guiding the network to learn channel and spatial feature information is crucial. This application introduces a dual-parallel attention feature fusion mechanism to replace the ordinary channel cascading method, fully integrating extracted features containing multi-scale contextual information. The dual-parallel attention feature fusion module progressively adjusts the weights of the feature maps over multiple iterations, allowing the network to repeatedly refine and optimize feature representations until the optimal fusion effect is achieved. By optimizing the feature fusion process, the dual-parallel attention feature fusion module can better capture the detailed features of the target object, thereby improving fusion accuracy.

[0066] The dual-parallel attention feature fusion module includes a first attention module and a second attention module. The first attention module is a parallel fusion of global and local attention, allowing it to simultaneously focus on global features and local details of the remote sensing image. The second attention module is a parallel fusion of channel attention and spatial attention. After fully fusing multi-scale features at both the global and local levels, channel attention and spatial attention are connected in parallel to enhance the fusion effect from both channel and spatial perspectives.

[0067] Specifically, the first attention module is based on spatial features and time characteristics The initial fusion features are obtained, and the second attention module is based on the spatial features. Time characteristics The initial fusion features yield intermediate features for image reconstruction. The specific feature fusion process is shown in equations (1) and (2):

[0068]

[0069]

[0070] In the formula, This represents the spatiotemporal fusion of the multi-scale spatial features and the multi-scale temporal features of the i-th layer. The local attention weights represent the features after fusion. Let z represent the global attention weights of the fused features, z represent the initial fused features obtained through the parallel structure of global and local attention, and M represent the global attention weights. C (z) represents the channel attention weights for the initial fusion features, M S (z) represents the spatial attention weights of the initial fusion features. This indicates that the dual-parallel attention feature fusion module integrates multi-scale features extracted from high- and low-resolution remote sensing images, including global, local, channel, and spatial attention, which are used as intermediate features for image reconstruction. The local attention weight, global attention weight, channel attention weight, and spatial attention weight are all real numbers between 0 and 1. Figure 3 The dashed part on the left side of the expression (1) represents the part in the expression (1). Calculate, the dotted part on the right represents (1-(M) in equation (2). C (z)+M S (z)))Calculation.

[0071] Step S4, intermediate features The image is input into an image reconstruction network to obtain a spatiotemporal fused remote sensing image at time t2.

[0072] The image reconstruction network can include multiple convolutional layers, deconvolutional layers, upsampling layers, downsampling layers, and possible residual connections. Through a series of nonlinear transformations and mappings, it generates a spatiotemporal fused remote sensing image at time t2. To achieve the reconstruction of image content.

[0073] Furthermore, during the training phase of the generative adversarial network, it is necessary to further integrate the spatiotemporally fused remote sensing images at time t2. The input image is fed to a discriminator, which determines whether the input image is a real observation image or a fused image. The discriminator employs a simple CNN structure, with each block consisting of multiple convolutional layers, a spectral normalization function (SpectralNorm), and a Leaky Rectified Linear Unit (LeakyReLU) activation function. The spectral normalization function stabilizes the adversarial process, while the convolutional operations maintain a 70×70 receptive field size.

[0074] Specifically, when the discriminator's input is the reference remote sensing image F2 at time t2 and the predicted remote sensing image C2 at time t2, the discriminator is expected to produce a true matrix, i.e., determine that the input image is the actual observed image. Conversely, when the discriminator's input is the spatiotemporally fused remote sensing image at time t2... When the predicted remote sensing image C2 is obtained at time t2, the discriminator is expected to produce a completely false matrix, thus determining that the input image is a fused image. The adversarial signal provided by the discriminator (i.e., its output matrix) is used as part of the generator's loss function, incentivizing the generator to produce images that are more difficult for the discriminator to distinguish, i.e., higher-quality fused images. By training the generator and discriminator alternately, the game between them gradually reaches an equilibrium point, at which point the generator can produce fused images that are close to the truth.

[0075] The spatiotemporal fusion method for remote sensing images based on generative adversarial networks (GANs) in this embodiment introduces a switchable multi-scale convolutional attention network and a dual parallel attention feature fusion module into the generator of the GAN. The switchable multi-scale convolutional attention network can adaptively adjust feature extraction at different scales, enabling the model to effectively capture key information even with limited reference images, significantly reducing the requirement for the number of reference images. At the same time, the dual parallel attention feature fusion module includes a first attention module that fuses global attention and local attention in parallel and a second attention module that fuses channel attention and spatial attention in parallel. The first attention module generates preliminary fusion features by capturing global trends and local details in the image, enhancing the richness and accuracy of feature representation. The second attention module further utilizes the spatial and temporal features of the remote sensing image, as well as the preliminary fusion features, and leverages the parallel enhancement of channel and spatial attention to deeply explore the intrinsic connections and importance differences between features, thereby generating more refined and discriminative intermediate features. Therefore, this method reduces the dependence on the number of reference images and improves the efficiency and accuracy of spatiotemporal fusion of remote sensing images by optimizing the image feature extraction process using a switchable multi-scale convolutional attention network and optimizing the feature fusion process using a dual parallel attention feature fusion module.

[0076] In one embodiment, the switchable multi-scale convolutional attention network includes at least one downsampled multi-scale convolutional attention block, each of which includes a switchable normalization layer (SwitchNorm), a multi-scale convolutional attention layer (Attention), and a feedforward network layer (FFN). The multi-scale convolutional attention layer includes depthwise convolution, multi-branch depthwise strip convolution, and 1×1 convolution.

[0077] Specifically, the downsampling multi-scale convolutional attention block first aggregates local information through ordinary depthwise convolution, and then uses multi-branch depthwise strip convolution to capture contextual information at different scales, with kernel sizes of 5, 7, and 11 for each branch. In particular, to reduce computational burden, depthwise strip convolution is used to simulate the effect of standard 2D convolution, such as a pair of 5×1 and 1×5 convolutions replacing the 5×5 convolution. Then, the contextual information at different scales captured by the multi-branch depthwise strip convolution is fused with the local information aggregated by ordinary depthwise convolution, and the 1×1 convolution is used to adjust the number of channels in the fused image and model the relationships between channels. This multi-scale design effectively aggregates rich semantic information, reduces redundancy in the self-attention mechanism, and improves efficiency. By enhancing salient regions in the image, the downsampling multi-scale convolutional attention block fully extracts feature context, which is beneficial for subsequent image reconstruction.

[0078] In downsampling multi-scale convolutional attention blocks, the FFN follows the multi-scale convolutional attention layer, playing a role similar to the non-linear transformation component in a Transformer structure. Specifically, the FFN internally employs a series of convolutional layers, such as... Figure 2 The 1×1 and 3×3 convolutions in the model perform linear mappings across different dimensions and introduce necessary non-linearity through non-linear activation functions, such as LeakyReLU, to help the model capture higher-order interactions in the data. This structural design allows FFN to increase the model's expressiveness, enabling it to describe and distinguish more complex and diverse data patterns. Simultaneously, the introduction of FFN enhances the model's learning potential, allowing it to exhibit higher adaptability and generalization ability when facing complex tasks.

[0079] In this embodiment, the multi-branch deep strip convolution can filter and emphasize important features, while the FFN network enhances the expressiveness of these features by introducing nonlinearity. Together, they promote a deeper understanding of the input data and high-quality feature representation, reduce the dependence on the number of reference images, and improve the reliability of feature fusion.

[0080] In one embodiment, such as Figure 3 As shown, local attention consists of two 1×1 convolutional layers, and global attention consists of a global average pooling layer and two 1×1 convolutional layers.

[0081] The input to the first 1×1 convolutional layer of the local attention layer is spatial features. With time characteristics The fused feature map. The output features of the first 1×1 convolutional layer are processed by batch normalization (BN) and ReLU activation before being input into the second 1×1 convolutional layer. The output of the second 1×1 convolutional layer is then processed by batch normalization again.

[0082] Specifically, the first 1×1 convolutional layer extracts preliminary feature representations by changing the depth (i.e., the number of channels) of the feature map while maintaining the spatial dimension of the input feature map. For example... Figure 3 As shown in the figure In this context, C represents the number of channels in the input feature map, and r represents the channel compression ratio, which is usually an integer greater than 1. This means that after the first 1×1 convolutional layer, the number of channels in the input feature map is reduced to [the original number]. The purpose of channel compression is to reduce computation and the number of parameters. The second 1×1 convolutional layer is used to expand the number of channels in the input feature map to the original number of channels C.

[0083] The input to the global average pooling layer is spatial features. With time characteristics The fused feature map. Global attention first averages the input feature map spatially using a global average pooling layer, resulting in an output feature map (C×1×1) with the same number of channels as the input feature map but a 1×1 spatial dimension. This process effectively compresses the information of each channel into a global representation that reflects the average activation level of that channel across the entire image. Next, the output feature map of the global average pooling layer is fed into the first 1×1 convolutional layer. This first 1×1 convolutional layer extracts preliminary feature representations by changing the depth (i.e., the number of channels) of the feature map while maintaining the spatial dimension of the input feature map, thus obtaining the output feature map. The output feature map of the first 1×1 convolutional layer is batch normalized and ReLU activated before being input into the second 1×1 convolutional layer. The second 1×1 convolutional layer restores the number of channels of the input feature map to C. Then, the output feature map (C×1×1) of the second 1×1 convolutional layer is batch normalized.

[0084] Finally, after obtaining the feature representations for local and global attention, feature maps are added element by element to combine the features processed by global and local attention with the spatial features. and time characteristics These features are combined to obtain preliminary fusion features. These preliminary fusion features contain both local details of the image and global contextual information, providing rich feature representations.

[0085] The local attention in this embodiment includes two 1×1 convolutional layers. By stacking the two 1×1 convolutional layers, the local attention module can focus on local regions in the image and extract rich detailed features. The global attention includes a global average pooling layer and two 1×1 convolutional layers. The global average pooling layer compresses the feature map in the spatial dimension and extracts global feature representations. Then, the two 1×1 convolutional layers transform and enhance these global features, which helps the model to grasp the global context of the image, thus making it more robust when dealing with complex scenes and improving the reliability of spatiotemporal fusion of remote sensing images.

[0086] The goal of spatial attention is to emphasize important spatial locations in the input feature map while suppressing less important regions. This helps the model focus more on the key parts of an object, improving the accuracy of recognition and classification.

[0087] In one embodiment, spatial attention comprises two 1×1 convolutional layers and two dilated convolutional layers.

[0088] The input to the first 1×1 convolutional layer is the preliminary fused feature obtained through local and global attention. The main function of this first 1×1 convolutional layer is to compress the number of channels in the input feature map, reducing it to 1 / r of the original number of channels to reduce computation while preserving sufficient spatial information for subsequent processing. The output feature map of the first 1×1 convolutional layer, after batch normalization and ReLU activation, is input to the first dilated convolutional layer (Dialition_Conv3). The output feature map of the first dilated convolutional layer... After batch normalization and ReLU activation, the data is fed into the second dilated convolutional layer (Dialition_Conv3). Similarly, the output feature map of the second dilated convolutional layer... After batch normalization and ReLU activation, the data is fed into a second 1×1 convolutional layer. The second 1×1 convolutional layer reduces the number of channels to 1, generating a spatial attention map (1×H×W). This spatial attention map is a single-channel map with the same spatial size as the initial fused features, and each value in the spatial attention map represents the importance weight of the corresponding location.

[0089] In this embodiment, the spatial attention mechanism first performs efficient channel compression on the input feature map using a first 1×1 convolutional layer, reducing the computational cost of subsequent layers. Then, two dilated convolutional layers increase the effective receptive field of the convolutional kernels, enabling the model to capture broader contextual information and long-range spatial dependencies without adding additional parameters. This spatial attention not only enhances the model's understanding and representation of complex remote sensing image features but also significantly improves the reliability and efficiency of spatiotemporal fusion tasks.

[0090] Dilated convolutional layers increase the effective receptive field of the convolutional kernel by introducing a dilatation rate, while keeping the number of parameters and computational cost relatively constant. The dilatation rate determines the sampling interval of the convolutional kernel on the input data, that is, the distance between adjacent elements.

[0091] In one embodiment, the dilatational convolutional layer has a dilatational rate of 4. When the dilatational rate is set to 4, it means that a 3×3 zero-value region is formed around each element of the convolutional kernel. In this case, the actual sampling points of the convolutional kernel on the input data become sparser, but each sampling can cover input data points further away, thereby significantly increasing the receptive field without increasing the physical size of the convolutional kernel.

[0092] In one embodiment, such as Figure 3 As shown, the channel attention includes an average pooling branch and a max pooling branch, the average pooling branch and the max pooling branch are connected in parallel, the average pooling branch includes an average pooling layer and a first fully connected layer, and the max pooling branch includes a max pooling layer and a second fully connected layer.

[0093] The channel attention mechanism comprises two parallel processing branches, employing max pooling and average pooling methods respectively to obtain global statistical features of the initial fused features. For the average pooling branch, a global average pooling operation is performed on the initial fused features through an average pooling layer. Global average pooling calculates the average value for each channel of the feature map, resulting in a feature vector with the same number of channels as the initial fused features but a spatial dimension (height and width) of 1. This feature vector reflects the average feature strength of each channel. For the max pooling branch, a global max pooling operation is performed on the initial fused features through a max pooling layer. Similar to average pooling, global max pooling also operates on each channel individually, but it calculates the maximum value for each channel, also resulting in a feature vector with a spatial dimension of 1. This vector reflects the most salient feature of each channel. After obtaining the global statistical features, both branches perform feature transformation and dependency modeling through one or more fully connected layers (FC). For example... Figure 3In this approach, each of the two branches generates its own channel features through two fully connected layers. These channel features are then summed element-wise to generate the final channel attention weights. Each element in the channel attention weights can be considered the optimal weight for its corresponding feature channel.

[0094] Furthermore, such as Figure 3 As shown, the output of the fully connected layer can also be batch normalized to accelerate the training process and reduce overfitting, thereby improving the efficiency of spatiotemporal fusion of remote sensing images. Furthermore, the ReLU activation function can be applied to the output of the fully connected layer after batch normalization to introduce nonlinearity, thereby enhancing the expressive power of the model and improving the reliability of spatiotemporal fusion of remote sensing images.

[0095] The channel attention in this embodiment includes two parallel processing branches: max pooling and average pooling. By using max pooling and average pooling in parallel to obtain global statistical information of the input feature map, the model can simultaneously consider the most salient features and average features in the feature channels. This makes the model more comprehensive and accurate in judging the importance of feature channels, and further improves the reliability of spatiotemporal fusion of remote sensing images.

[0096] In one embodiment, the remote sensing image spatiotemporal fusion method further includes:

[0097] Step S51: Combine the spatiotemporal fused remote sensing images at time t2. The input is sent to the discriminator for identification, and the identification result is obtained.

[0098] In this context, the spatiotemporal fused remote sensing image at time t2 is the spatiotemporal fused remote sensing image at time t2 generated by the generator in the conditional generative adversarial network. The discriminator's task is to distinguish whether the input image is the real remote sensing image F2 at time t2 or the spatiotemporal fused remote sensing image generated by the generator. Through training, the discriminator can learn the difference features between real remote sensing images and fused images, thereby providing a discrimination result.

[0099] Step S52: Based on the identification results, the reference remote sensing image F1 at time t1, the predicted remote sensing image C2 at time t2, and the spatiotemporally fused remote sensing image at time t2. Calculate the value of the loss function.

[0100] The loss function can include pixel-level loss (such as L1 or L2 loss) between the generated and real images, adversarial loss, and structural loss. Pixel-level loss directly compares the values ​​of the two images at corresponding pixel locations and calculates the difference between them. Adversarial loss encourages the generated image to deceive the discriminator, typically by maximizing the discriminator's probability of misclassifying the generated image. Structural loss ensures that the generated fused image retains important features while maintaining a structure similar to the original image.

[0101] Step S53: Update the parameters of the switchable multi-scale convolutional attention network, the dual parallel attention feature fusion module, and the discriminator based on the value of the loss function.

[0102] Specifically, based on the calculated loss function value, the backpropagation algorithm is used to update the parameters of the switchable multi-scale convolutional attention network, the dual parallel attention feature fusion module, and the discriminator. This process is iterative and aims to minimize the loss function, so that the generated spatiotemporal fused remote sensing image at time t2 is visually closer to the real remote sensing image at time t2.

[0103] Step S54: Repeat steps S51 to S53 until the value of the loss function meets the preset conditions, and use the parameters of the current switchable multi-scale convolutional attention network, dual parallel attention feature fusion module and discriminator as the final determined model parameters.

[0104] The preset conditions may include the loss function value falling below a certain threshold, the loss value no longer decreasing significantly, or the number of iterations reaching an upper limit. When the loss function value meets the preset conditions, the model is considered to have converged, and the parameters of the current switchable multi-scale convolutional attention network, dual parallel attention feature fusion module, and discriminator are used as the final model parameters. These parameters will be used to generate high-quality spatiotemporal fused remote sensing images.

[0105] This embodiment improves the spatiotemporal fusion efficiency and reliability of remote sensing images by iteratively training the generative adversarial network and optimizing the model parameters.

[0106] In one embodiment, the loss function includes a total loss function for the conditional generation network and a discriminator loss function. The total loss function for the conditional generation network... The expression for is shown in equation (3):

[0107]

[0108] In the formula, This represents the loss function of a least-squares generative adversarial network. This represents the L1 loss function, which calculates the sum of the absolute differences between corresponding pixels in the generated image and the real image. It is less sensitive to outliers, i.e., points far from the average, and helps generate sharper image edges. Represents the spectral loss function. Let α represent the structural loss function, α represent the weights of the least squares generative adversarial network loss function, and β represent... The weights of the loss functions, where γ represents the weight of the spectral loss function and δ represents the weight of the structural loss function.

[0109] Least squares generative adversarial network loss function The specific expression is shown in equation (4):

[0110]

[0111] In the formula, F1 represents the reference remote sensing image at time t1, C2 represents the predicted remote sensing image at time t2, E represents the expectation, and G(F1, C2) represents the spatiotemporal fused remote sensing image at time t2 obtained by inputting the reference remote sensing image F1 at time t1 and the predicted remote sensing image C2 at time t2 into the generator. D(G(F1, C2), C2) represents the combination of the predicted remote sensing image C2 at time t2 and the spatiotemporally fused remote sensing image at time t2. The identification result obtained after inputting into the discriminator.

[0112] The specific expression for the loss function is shown in equation (5):

[0113]

[0114] In the formula, K represents the number of pixels in the remote sensing image, T represents the real image matrix, P represents the predicted image matrix, and ||.||1 represents the L1 norm.

[0115] Spectral loss function The specific expression is shown in equation (6):

[0116]

[0117] In the formula, I is a tensor of all 1s, and ||.||2 represents the L2 norm.

[0118] Structural loss function The specific expression is shown in equation (7):

[0119]

[0120] In the formula, l m V represents the brightness comparison at the j-th scale. NLet represent the weighting coefficients for brightness comparison at the j-th scale, n represent the number of highest-level scales, and c represent the weighting coefficients. j ρ represents the contrast comparison at the j-th scale. j s represents the weighting coefficient for contrast comparison at the j-th scale. j μ represents the structural comparison at the j-th scale. j This represents the weighting coefficient for structural comparison at the j-th scale.

[0121] Discriminator loss function The specific expression is shown in equation (8):

[0122]

[0123] In the formula, F2 represents the true high spatial resolution remote sensing image at time t2, and D(F2, C2) represents the discrimination result obtained after inputting the predicted remote sensing image C2 at time t2 and the true high spatial resolution remote sensing image F2 at time t2 into the discriminator.

[0124] In this embodiment, the generator's loss function, in addition to the traditional least-squares generative adversarial network loss function, also incorporates an L1 loss function, a spectral loss function, and a structural loss function to ensure the realism and accuracy of the generated images. The L1 loss function ensures that the generated images are consistent with the real images at the pixel level, while the spectral and structural loss functions focus on accurately capturing spectral similarity and multi-scale structural similarity, respectively, further improving the detail richness and overall quality of the generated images. Furthermore, by assigning flexible and adjustable weights to the four loss functions, the model can be made more flexible, thereby meeting the specific needs and standards of spatiotemporal fusion of remote sensing images in different application scenarios.

[0125] In one embodiment, the image reconstruction network includes an upsampling residual module, which upsamples intermediate features through transposed convolutions and skip connections to obtain the spatiotemporal fused remote sensing image at time t2.

[0126] The image reconstruction network transforms processed feature information into a high-quality spatiotemporal fused remote sensing image. The input to the image reconstruction stage is the feature data processed by the dual-parallel attention feature fusion module. During reconstruction, each upsampled residual block receives feature input from the previous layer and is upsampled through transposed convolution. This upsampled feature is then combined with multi-level features containing multi-scale contextual information introduced in the feature extraction stage via skip connections to generate a larger, more detailed feature map. The final spatiotemporal fused remote sensing image at time t2 is generated. It is obtained by constructing and fusing multi-layer upsampling residual blocks layer by layer. It inherits feature information from different layers, thus presenting higher spatial resolution and richer detail information.

[0127] In this embodiment, the image reconstruction network increases the size of the feature map by transposing convolutions and combines skip connections with multi-level features containing multi-scale contextual information from the feature extraction stage, thereby improving the accuracy and reliability of image reconstruction.

[0128] To better understand the spatiotemporal fusion method for remote sensing images based on generative adversarial networks in the above embodiments, a more detailed embodiment is provided below for explanation.

[0129] In one embodiment, the specific structure of the generative adversarial network (GAN) in the spatiotemporal fusion method for remote sensing images is as follows: Figure 4 As shown, the method includes a generator and a discriminator. The generator comprises an image feature extraction network, an image feature fusion network, and an image reconstruction network. This method incorporates a multi-scale convolutional attention mechanism and a dual-parallel attention feature fusion mechanism into the image feature extraction network. Simultaneously, residual connections are introduced in both feature extraction and image reconstruction processes to preserve multi-level, multi-scale contextual features during feature extraction. Skip connections used in image reconstruction effectively avoid gradient vanishing and gradient exploding, while compensating for spatial resolution loss. This method requires only one high-temporal-resolution, low-spatial-resolution remote sensing image and one low-temporal-resolution, high-spatial-resolution remote sensing image to generate a high-quality fused image, solving the problem of existing methods requiring a large number of reference images.

[0130] The image feature extraction network consists of an F1 image feature extraction network and a C2 image feature extraction network. The F1 network extracts spatial features at different scales from the reference remote sensing image F1 at time t1. The C2 network extracts temporal features at different scales from the predicted remote sensing image C2 at time t2. Both the F1 and C2 networks include a cascaded 1×1 convolutional layer and four downsampled multi-scale convolutional attention blocks. Each downsampled multi-scale convolutional attention block includes a switchable multi-scale convolutional attention network (SMSCAN), a 3×3 convolutional layer, and a ReLU activation layer. The specific structure of the downsampled multi-scale convolutional attention block is as follows: Figure 2 As shown. The image feature fusion network includes a dual parallel attention feature fusion module (DPAFF), the specific structure of which is as follows. Figure 3 As shown.

[0131] The image reconstruction network consists of four cascaded 3×3 transposed convolutional layers and one 1×1 convolutional layer, with ReLU activation applied to the output of each transposed and convolutional layer. The input to the first 3×3 transposed convolutional layer includes intermediate features from the output of a dual parallel attention feature fusion module. The input to the first 3×3 transposed convolutional layer in the F1 image feature extraction network includes the output of the first 3×3 transposed convolutional layer and the output of the first feature extraction module in the F1 image feature extraction network. The input to the third 3×3 transposed convolutional layer includes the output of the second 3×3 transposed convolutional layer and the output of the second feature extraction module in the F1 image feature extraction network. The input to the fourth 3×3 transposed convolutional layer includes the output of the third 3×3 transposed convolutional layer and the output of the third feature extraction module in the F1 image feature extraction network. The input to the 1×1 convolutional layer includes the output of the fourth 3×3 transposed convolutional layer and the output of the fourth feature extraction module in the F1 image feature extraction network.

[0132] The discriminator employs a simple CNN architecture. Each block of the discriminator consists of multiple convolutional layers, spectral normalization (SpectralNorm), and Leaky Rectified Linear Unit (LeakyReLU) activation functions. Spectral normalization stabilizes the adversarial process, while the convolutional operations maintain a 70×70 receptive field size. During the discriminator's training phase, when its input is F2 and C2, it is expected to produce a true matrix. Conversely, when its input is... When using C2, the goal is to produce a completely fake matrix. The adversarial function provided by the discriminator helps the generator achieve higher quality results. Specifically, such as... Figure 4 As shown, the discriminator consists of a cascaded 1×1 convolutional layer, four spectral normalization modules, another 1×1 convolutional layer, and a PatchMap layer.

[0133] To verify the effectiveness of the method, the generative adversarial network in this method was used as the spatiotemporal fusion model for remote sensing images in this embodiment. Related experiments were conducted on two publicly available remote sensing datasets, CIA and LGC, as follows:

[0134] Quantitative experiment:

[0135] This embodiment selected four models: STARFM (Spatial and Temporal Adaptive Reflectance Fusion Model), FSDAF (Flexible Spatiotemporal Data Fusion), DCSTFN (Deep Convolutional Spatiotemporal Fusion Network), and GANSTFM (Generative Adversarial Network-based Spatiotemporal Fusion Model). Comparative experiments were conducted with the remote sensing image spatiotemporal fusion model of this embodiment on the publicly available remote sensing datasets CIA and LGC. STARFM and FSDAF are traditional methods with good performance, while DCSTFN and GANSTFM are classic deep learning-based methods. DCSTFN uses a CNN framework, and GANSTFM uses a GAN architecture. Specific quantitative analysis of the experimental results on the CIA dataset is shown in Table 1.

[0136] Table 1. Experimental results related to the CIA dataset.

[0137]

[0138]

[0139] The comparative experiments comprehensively evaluated the model's performance using six commonly used evaluation metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Spectral Angle Mapper (SAM), Relative Dimensionless Global Synthesis Error (ERGAS), Spatial Correlation Coefficient (CC), and Root Mean Square Error (RMSE). Higher SSIM, PSNR, and CC values ​​indicate greater similarity between the spatiotemporally fused image and the real image, while lower RMSE, SAM, and ERGAS values ​​indicate better spatiotemporal fusion performance and model stability. Experimental results show the performance and average values ​​of the six bands of remote sensing images on SSIM, PSNR, CC, and RMSE, as well as the results on SAM and ERGAS. The final results demonstrate that the proposed spatiotemporal fusion model for remote sensing images achieves optimal values ​​across almost all metrics. For example, in the CIA dataset, regarding the SSIM index, which is related to image structural similarity, the spatiotemporal fusion model of this embodiment outperforms the traditional STARFM and FSDAF methods by approximately 10.1% and 1.9%, respectively. Compared to deep learning-based DCSTFN and GANSTFM methods, the spatiotemporal fusion model in this embodiment achieves approximately 1.2% and 1.5% improvement in SSIM, respectively. On the LGC dataset, the spatiotemporal fusion model in this embodiment outperforms the traditional STARFM and FSDAF methods by approximately 10.8% and 4.2%, respectively. Compared to deep learning-based DCSTFN and GANSTFM methods, the spatiotemporal fusion model in this embodiment achieves approximately 3.3% and 1.6%, respectively. We attribute these improvements to several factors. First, the powerful model performance of generative adversarial networks produces clearer and more realistic samples. Second, the remote sensing image spatiotemporal fusion model in this embodiment incorporates multi-scale convolutional attention (SMSCAN), which effectively handles spatial and distance dependencies in images while capturing detailed features at multiple scales. Furthermore, attention weight allocation based on multiple scales allows for greater attention to more important multi-scale features. In the feature fusion stage, a dual parallel attention feature fusion module (DPAFF) is used to fully fuse the extracted multi-scale features. By assigning different attention weights to the contributions of the multi-scale features obtained in the feature extraction stage to the final output, excellent fusion results are achieved.

[0140] The specific quantitative analysis of the experimental results for the LGC dataset is shown in Table 2.

[0141] Table 2 Experimental results related to the LGC dataset

[0142]

[0143]

[0144] As shown in Table 2, the spatiotemporal fusion model of this embodiment is significantly better than other models in terms of PSNR and SSIM, which indicates that the spatiotemporal fusion model of this embodiment extracts structural information and texture details more effectively. Figure 8 A heatmap comparing the prediction results of each model in the LGC dataset was presented. The plot shows the high "point cloud" density of all models, a characteristic of the LGC dataset. However, higher model density implies higher image resolution during reconstruction, which may enhance the target detection capability of the final remote sensing image. Compared to other models, the scatter points of the spatiotemporal fusion model in this embodiment are densely concentrated near the diagonal. Overall, the scatter point distribution of the spatiotemporal fusion model in this embodiment is closely aligned with the diagonal across all bands, with bright areas concentrated near the diagonal, indicating that the prediction results of the spatiotemporal fusion model in this embodiment are closer to the actual observations. This also demonstrates the robustness of the spatiotemporal fusion method for remote sensing images in this embodiment when dealing with complex variations.

[0145] Qualitative experiment

[0146] To visually represent our experimental results, Figure 5 Quantitative experimental results of STARFM, FSDAF, DCSTFN, GANSTFM, and the remote sensing image spatiotemporal fusion model (OURS) of this embodiment are shown on the CIA and LGC datasets. The CIA dataset mainly contains crop phenological data, including many farmland planting and irrigation areas. A color image composed of bands 3, 4, and 5 is selected for visual demonstration. Figure 5 The first row is an overview of the spatiotemporal fusion images generated by each model. "(a)Observed" is the actual observed image, "(b)STARFM" is the spatiotemporal fusion image generated by STARFM, "(c)FSDAF" is the spatiotemporal fusion image generated by FSDAF, "(d)DCSTFN" is the spatiotemporal fusion image generated by DCSTFN, "(e)GANSTFM" is the spatiotemporal fusion image generated by GANSTFM, and "(f)OURS" is the spatiotemporal fusion image generated by the remote sensing image spatiotemporal fusion model in this embodiment. Figure 5The second row shows a magnified representation of the image within the yellow box in the overall effect image of the first row (Zoomed Details). As can be seen from the overall effect image, the spatiotemporal fusion image generated by the remote sensing image spatiotemporal fusion model in this embodiment has a clarity closer to the original image. STARFM and FSDAF perform poorly in feature-dense regions, mainly because they focus on spatiotemporal features while ignoring the details of texture features. Deep learning-based methods such as DCSTFN and GANSTFM can not only predict dense regions but also clearly identify differences between different color regions, accurately predicting information such as farmland and roads. However, the results show that the remote sensing image spatiotemporal fusion model in this embodiment exhibits better performance in irregular regions. Furthermore, Figure 7 The diagram shows a heat map of the prediction results from various models in the CIA dataset, within the same band. The horizontal axis of the heat map represents the band of the predicted image, and the vertical axis represents the band of the actual image. Heat maps are primarily used for target localization and counting. It can be seen that, compared to other models, the spatiotemporal fusion model of remote sensing images in this embodiment has a very dense scatter plot near the diagonal, with bright areas concentrated near the diagonal. This indicates that the prediction results of the spatiotemporal fusion model of remote sensing images in this embodiment are closer to the actual observations and have stronger robustness when dealing with complex changes.

[0147] Figure 6 The fusion results of the various models on the LGC dataset are shown. Figure 6 As shown, the output image of the STARFM model contains excessive noise. This problem arises because STARFM, as an earlier spatiotemporal fusion algorithm, lacks the ability to detect spatial features of region boundaries (such as farmland and roads). The output image of the FSDAF model exhibits significant bias within the region because the algorithm cannot adapt to dynamic features and repeatedly segments unchanged pixels. In contrast, the spatiotemporal fusion image generated by the remote sensing image spatiotemporal fusion model in this embodiment is very close to the actual image, with minimal overall difference. This accuracy is attributed to the inclusion of multi-scale convolutional attention, which captures detailed features across multiple scales while effectively managing spatial and long-range dependencies in the image.

[0148] The spatiotemporal fusion method for remote sensing images based on generative adversarial networks in this embodiment introduces a multi-scale convolutional attention mechanism into the feature extraction network. This addresses the problem of previous multi-scale mechanisms failing to distinguish the importance of multi-scale features without significantly increasing computational cost. When extracting multi-scale contextual information, it assigns different attention weights to features of different scales according to their importance, enhancing salient regions. A dual-parallel attention feature fusion module is introduced into the feature fusion process, primarily to optimize the fusion of multi-scale feature maps. Compared to ordinary attention mechanisms, the dual-parallel attention feature fusion module progressively adjusts the weights of feature maps across multiple parallel attention structures, allowing the network to repeatedly correct and optimize feature representations until the optimal fusion effect is achieved. It focuses on solving the multi-scale feature fusion problem and is particularly suitable for processing images containing targets of various sizes. By optimizing the feature fusion process, the dual-parallel attention feature fusion module can better capture the detailed features of the target object, thereby improving fusion accuracy. It can adaptively adjust attention according to the size and shape of the target, enabling the model to respond well to targets of different scales.

[0149] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0150] Based on the same inventive concept, this application also provides a generative adversarial network (GAN)-based remote sensing image spatiotemporal fusion apparatus for implementing the aforementioned GAN-based remote sensing image spatiotemporal fusion method. The solution provided by this apparatus is similar to the implementation described in the above method. Therefore, the specific limitations of one or more GAN-based remote sensing image spatiotemporal fusion apparatus embodiments provided below can be found in the limitations of the GAN-based remote sensing image spatiotemporal fusion method described above, and will not be repeated here.

[0151] In one embodiment, such as Figure 9 As shown, a spatiotemporal fusion device 900 for remote sensing images based on generative adversarial networks is provided, comprising: a remote sensing image acquisition module 901, an image feature extraction module 902, an image feature fusion module 903, and a spatiotemporal fusion remote sensing image generation module 904, wherein:

[0152] The remote sensing image acquisition module 901 is used to acquire the reference remote sensing image F1 at time t1 and the predicted remote sensing image C2 at time t2; the reference remote sensing image at time t1 is a remote sensing image with low temporal resolution and high spatial resolution, and the predicted remote sensing image at time t2 is a remote sensing image with high temporal resolution and low spatial resolution.

[0153] Image feature extraction module 902 is used to extract spatial features at different scales of the reference remote sensing image F1 at time t1 and temporal features at different scales of the predicted remote sensing image C2 at time t2 using a switchable multi-scale convolutional attention network, to obtain a set of spatial features. and a set of time features

[0154] Image feature fusion module 903 is used to fuse spatial features and time characteristics Feature fusion is performed using a dual parallel attention feature fusion module to obtain intermediate features for image reconstruction. The dual-parallel attention feature fusion module includes a first attention module and a second attention module. The first attention module is a parallel fusion of global attention and local attention, while the second attention module is a parallel fusion of channel attention and spatial attention. The first attention module is based on spatial features... and time characteristics The initial fusion features are obtained, and the second attention module is based on the spatial features. Time characteristics Intermediate features are obtained from the initial fusion features.

[0155] The spatiotemporal fusion remote sensing image generation module 904 generates intermediate features. The image is input into an image reconstruction network to obtain a spatiotemporal fused remote sensing image at time t2.

[0156] In one embodiment, the remote sensing image spatiotemporal fusion device 900 based on generative adversarial networks is further used to fuse the spatiotemporal remote sensing images at time t2. The data is input to a discriminator for identification, and the identification result is obtained. Based on the identification result, the reference remote sensing image F1 at time t1, the predicted remote sensing image C2 at time t2, and the spatiotemporally fused remote sensing image at time t2... Calculate the value of the loss function; update the parameters of the switchable multi-scale convolutional attention network, the dual parallel attention feature fusion module, and the discriminator based on the value of the loss function; repeat the above steps until the value of the loss function meets the preset conditions, and use the current parameters of the switchable multi-scale convolutional attention network, the dual parallel attention feature fusion module, and the discriminator as the final determined model parameters.

[0157] The modules in the aforementioned spatiotemporal fusion device for remote sensing images based on generative adversarial networks can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0158] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0160] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0161] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0162] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A spatio-temporal fusion method for remote sensing images based on a generative adversarial network, characterized in that, The generator in the generative adversarial network comprises a switchable multi-scale convolution attention network and a double-parallel attention feature fusion module, and the method comprises the following steps: In step S1, a reference remote sensing image F1 at time t1 and a predicted remote sensing image C2 at time t2 are obtained; the reference remote sensing image at time t1 is a remote sensing image with low temporal resolution and high spatial resolution, and the predicted remote sensing image at time t2 is a remote sensing image with high temporal resolution and low spatial resolution; Step S2, using the switchable multi-scale convolution attention network to extract the spatial features of different scales of the reference remote sensing image F1 at the t1 moment and the temporal features of different scales of the predicted remote sensing image C2 at the t2 moment respectively, to obtain a group of spatial features and a group of temporal features ; Step S3, fusing the spatial features and the temporal features to obtain intermediate features for image reconstruction ; the double-parallel attention feature fusion module comprises a first attention module and a second attention module, the first attention module is parallel fusion of global attention and local attention, and the second attention module is parallel fusion of channel attention and spatial attention; the first attention module obtains preliminary fusion features according to the spatial features and the temporal features , and the second attention module obtains the intermediate features according to the spatial features , the temporal features and the preliminary fusion features . Step S4, inputting the intermediate features to the image reconstruction network to obtain the spatio-temporal fusion remote sensing image at t2 ; The switchable multi-scale convolution attention network comprises at least one down-sampling multi-scale convolution attention block, each of the down-sampling multi-scale convolution attention blocks comprises a switchable normalization layer, a multi-scale convolution attention layer and a feedforward network layer, the multi-scale convolution attention layer comprises a depth convolution, a multi-branch depth strip convolution and a 1x1 convolution, and the kernel sizes of each branch in the multi-branch depth strip convolution are set to 5, 7 and 11 respectively.

2. The method of claim 1, wherein the method is based on a generative adversarial network. The local attention comprises two 1x1 convolution layers, and the global attention comprises a global average pooling layer and two 1x1 convolution layers. 3.The method of claim 1, wherein, The spatial attention comprises two 1x1 convolution layers and two dilated convolution layers.

4. The method of claim 3, wherein the method is based on a generative adversarial network. The dilated convolution layer has a dilation rate of 4.

5. The method of claim 1, wherein the method is based on a generative adversarial network. The channel attention comprises an average pooling branch and a maximum pooling branch, the average pooling branch and the maximum pooling branch are connected in parallel, the average pooling branch comprises an average pooling layer and a first full connection layer, and the maximum pooling branch comprises a maximum pooling layer and a second full connection layer.

6. The method of claim 1, wherein the method is based on a generative adversarial network. The method further comprises the following steps: Step S51, spatiotemporal fusion remote sensing image at t2 time is obtained The input is discriminated by the discriminator to obtain a discrimination result. Step S52, according to the identification result, the reference remote sensing image F1 at the t1 moment, the predicted remote sensing image C2 at the t2 moment and the space-time fusion remote sensing image calculating a value of the loss function; In step S53, the parameters of the switchable multi-scale convolution attention network, the double-parallel attention feature fusion module and the discriminator are updated according to the value of the loss function; In step S54, steps S51 to S53 are repeated until the value of the loss function meets a preset condition, and the current parameters of the switchable multi-scale convolution attention network, the double-parallel attention feature fusion module and the discriminator are taken as final determined model parameters.

7. The method of claim 6, wherein the method is based on a generative adversarial network. The loss function comprises a conditional generative network total loss function and a discriminator loss function; The conditional generation network total loss function The expression is: In the formula, denotes a least square generative adversarial network loss function, denotes an L1 loss function, denotes a spectral loss function, denotes a structure loss function, denotes a weight of the least square generative adversarial network loss function, denotes a weight of the L1 loss function, denotes a weight of the spectral loss function, denotes a weight of the structure loss function; wherein, The least square generative adversarial network loss function The expression is: In the formula, F1 represents the reference remote sensing image at the t1 moment, C2 represents the predicted remote sensing image at the t2 moment, E represents expectation, representing that the reference remote sensing image at the t1 moment , the predicted remote sensing image at the t2 moment input into the generator to obtain the spatio-temporal fusion remote sensing image at the t2 moment , representing that the predicted remote sensing image C2 at the t2 moment and the spatio-temporal fusion remote sensing image at the t2 moment input into the discriminator to obtain the discrimination result; The L1 loss function The expression is as follows: wherein denotes the number of pixels of the remote sensing image, denotes the real image matrix, denotes the predicted image matrix, denotes the LI norm; The spectral loss function The expression of the spectral loss function is as follows: wherein is a tensor of all ones, denotes the L2 norm; The structural loss function The expression of the structural loss function is as follows: In the formula, Indicates the first Brightness comparison at various scales Indicates the first The weighting coefficients for brightness comparison at each scale, where n represents the number of scales. This represents the contrast comparison at the j-th scale. This represents the weighting coefficient for the contrast comparison at the j-th scale. This represents a structural comparison at the j-th scale. Represents the weighting coefficients for structural comparisons at the j-th scale; The discriminator loss function The expression is: In the formula, a real high spatial resolution remote sensing image at time t2, a predicted remote sensing image C2 at time t2 and a real high spatial resolution remote sensing image at time t2 a discrimination result obtained after inputting to the discriminator. 8.The method of claim 1, wherein, The image reconstruction network comprises an up-sampling residual module, which performs up-sampling processing on the intermediate features by transposed convolution plus a jump connection to obtain the spatio-temporal fusion remote sensing image at the t2 moment .

9. A remote sensing image spatio-temporal fusion device based on a generative adversarial network, characterized in that, The device comprises: A remote sensing image acquisition module is configured to acquire a reference remote sensing image F1 at time t1 and a predicted remote sensing image C2 at time t2; the reference remote sensing image at time t1 is a remote sensing image with low temporal resolution and high spatial resolution, and the predicted remote sensing image at time t2 is a remote sensing image with high temporal resolution and low spatial resolution; The image feature extraction module is configured to extract spatial features of different scales of the reference remote sensing image F1 at the t1 moment and temporal features of different scales of the predicted remote sensing image C2 at the t2 moment respectively by using a switchable multi-scale convolution attention network, to obtain a group of spatial features and a group of temporal features The switchable multi-scale convolution attention network comprises at least one down-sampling multi-scale convolution attention block, each of the down-sampling multi-scale convolution attention blocks comprises a switchable normalization layer, a multi-scale convolution attention layer and a feedforward network layer, the multi-scale convolution attention layer comprises a depth convolution, a multi-branch depth strip convolution and a 1×1 convolution, and the size of the convolution kernel of each branch in the multi-branch depth strip convolution is set to 5, 7 and 11 respectively. Image feature fusion module, used to fuse the spatial features and time characteristics Feature fusion is performed using a dual parallel attention feature fusion module to obtain intermediate features for image reconstruction. The dual-parallel attention feature fusion module includes a first attention module and a second attention module. The first attention module is a parallel fusion of global attention and local attention, and the second attention module is a parallel fusion of channel attention and spatial attention. The first attention module, based on the spatial features... and the time features The initial fusion features are obtained, and the second attention module then uses these spatial features to... The time features The intermediate features are obtained by combining the initial fusion features. ; The spatio-temporal fusion remote sensing image generation module inputs the intermediate feature to the image reconstruction network to obtain a spatio-temporal fusion remote sensing image at t2 .

Images