Remote sensing image space-time fusion method based on generative adversarial network

Abstract

The invention relates to the technical field of remote sensing, in particular to a remote sensing image space-time fusion method and device based on a generative adversarial network, and the method comprises the steps: employing a switchable multi-scale convolution attention network to extract different-scale spatial features of a reference remote sensing image F1 at a t1 moment and different-scale time features of a prediction remote sensing image C2 at a t2 moment; performing feature fusion on the spatial features and the time features through a double-parallel attention feature fusion module to obtain intermediate features for image reconstruction, and inputting the intermediate features into an image reconstruction network to obtain a space-time fusion remote sensing image at the moment t2, according to the method, the dependence on the number of reference images is reduced, and the efficiency and accuracy of time-space fusion of the remote sensing image are improved.

Description

Technical Field

[0001] The present application relates to the field of remote sensing technology, and in particular to a method and device for spatiotemporal fusion of remote sensing images based on a generative adversarial network. Background Art

[0002] Remote sensing images with high spatial and temporal resolution play a vital role in natural disasters, vegetation cover, land use, and air pollution monitoring. However, such images are often not easy to obtain due to technical and cost limitations. Satellites with high spatial resolution usually have longer revisit periods. In contrast, satellites with short revisit periods usually provide lower spatial resolution. Space-time fusion technology provides a cost-effective method for obtaining remote sensing images with high spatial and temporal resolution. This technology combines the medium-resolution images of the MODIS sensor with the high-resolution images of the Landsat satellite to generate fused images with high spatial and temporal resolution.

[0003] Although the existing spatiotemporal fusion methods can achieve good fusion effects, the current mainstream methods mostly rely on four to five high-quality reference remote sensing images as input, and have high requirements for image quality and quantity. Taking the patent with publication number CN115731141A as an example, it proposes an innovative spatiotemporal fusion method for dynamic monitoring of mobile targets. This method is based on a trained spatiotemporal fusion network model and needs to integrate multiple low-resolution and high-resolution images from different data sources across multiple time points to accurately predict the high-resolution image at a specific moment. However, in practice, due to the influence of environmental factors, it is not easy to collect enough high-quality reference images, which directly limits the wide applicability and real-time performance of the method. Therefore, it is of great significance to reduce the model's requirements for the number of reference images. Summary of the invention

[0004] Based on this, it is necessary to provide a remote sensing image spatiotemporal fusion method and device based on a generative adversarial network that can reduce the model's requirements for the number of reference images in order to address the above technical problems.

[0005] In a first aspect, the present application provides a remote sensing image spatiotemporal fusion method based on a generative adversarial network, wherein the generator in the generative adversarial network includes a switchable multi-scale convolutional attention network and a dual parallel attention feature fusion module, and the method includes:

[0006] Step S1, obtaining a reference remote sensing image F1 at time t1 and a predicted remote sensing image C2 at time t2; the reference remote sensing image at time t1 is a remote sensing image with low temporal resolution and high spatial resolution, and the predicted remote sensing image at time t2 is a remote sensing image with high temporal resolution and low spatial resolution;

[0007] Step S2, a switchable multi-scale convolutional attention network is used to extract spatial features of different scales of the reference remote sensing image F1 at time t1 and temporal features of different scales of the predicted remote sensing image C2 at time t2, and obtain a set of spatial features and a set of temporal features

[0008] Step S3: transform the spatial features and time characteristics Feature fusion is performed through a dual parallel attention feature fusion module to obtain intermediate features for image reconstruction. The dual parallel attention feature fusion module includes a first attention module and a second attention module. The first attention module is a parallel fusion of global attention and local attention, and the second attention module is a parallel fusion of channel attention and spatial attention. The first attention module is based on the spatial features. and time characteristics After obtaining the preliminary fusion features, the second attention module is based on the spatial features Time characteristics And the initial fusion features get the intermediate features

[0009] Step S4: The intermediate features Input into the image reconstruction network to obtain the spatiotemporal fusion remote sensing image at time t2

[0010] In one embodiment, the switchable multi-scale convolutional attention network includes at least one downsampling multi-scale convolutional attention block, each downsampling multi-scale convolutional attention block includes a switchable normalization layer, a multi-scale convolutional attention layer and a feedforward network layer, the multi-scale convolutional attention layer includes deep convolution, multi-branch deep strip convolution and 1×1 convolution, and the convolution kernel size of each branch in the multi-branch deep strip convolution is set to 5, 7 and 11 respectively.

[0011] In one embodiment, the local attention includes two 1×1 convolutional layers, and the global attention includes a global average pooling layer and two 1×1 convolutional layers.

[0012] In one embodiment, the spatial attention includes two 1×1 convolutional layers and two dilated convolutional layers.

[0013] In one embodiment, the dilated convolutional layer has a dilation rate of 4.

[0014] In one embodiment, the channel attention includes an average pooling branch and a maximum pooling branch, the average pooling branch is connected in parallel with the maximum pooling branch, the average pooling branch includes an average pooling layer and a first fully connected layer, and the maximum pooling branch includes a maximum pooling layer and a second fully connected layer.

[0015] In one embodiment, the remote sensing image spatiotemporal fusion method based on a generative adversarial network further includes:

[0016] Step S51: fusion of the remote sensing image at time t2 Input to the discriminator for identification and obtain the identification result;

[0017] Step S52, based on the identification result, the reference remote sensing image F1 at time t1, the predicted remote sensing image C2 at time t2 and the spatiotemporal fusion remote sensing image at time t2 Calculate the value of the loss function;

[0018] Step S53, updating the parameters of the switchable multi-scale convolutional attention network, the dual parallel attention feature fusion module and the discriminator according to the value of the loss function;

[0019] Step S54, repeat steps S51 to S53 until the value of the loss function meets the preset conditions, and use the parameters of the current switchable multi-scale convolutional attention network, dual parallel attention feature fusion module and discriminator as the final model parameters.

[0020] In one embodiment, the loss function includes a conditional generative network total loss function and a discriminator loss function;

[0021] Conditional generation network total loss function The expression is:

[0022]

[0023] In the formula, represents the least squares generative adversarial network loss function, represents the L1 loss function, represents the spectral loss function, represents the structural loss function, α represents the weight of the least squares generative adversarial network loss function, β represents the weight of the L1 loss function, γ represents the weight of the spectral loss function, and δ represents the weight of the structural loss function; where,

[0024] Least Squares Generative Adversarial Network Loss Function The expression is:

[0025]

[0026] In the formula, F1 represents the reference remote sensing image at time t1, C2 represents the predicted remote sensing image at time t2, E represents the expectation, and G(F1, C2) represents the spatiotemporal fusion remote sensing image at time t2 obtained by inputting the reference remote sensing image F1 at time t1 and the predicted remote sensing image C2 at time t2 into the generator. D(G(F1, C2), C2) represents the spatiotemporal fusion of the predicted remote sensing image C2 at time t2 and the remote sensing image C2 at time t2. The identification result obtained after input into the discriminator;

[0027] L1 loss function The expression is as follows:

[0028]

[0029] Where K represents the number of pixels of the remote sensing image, T represents the real image matrix, P represents the predicted image matrix, and ||.||1 represents the L1 norm;

[0030] Spectral loss function The expression is as follows:

[0031]

[0032] Where I is a tensor of all 1s, ||.||2 represents the L2 norm;

[0033] Structural loss function The expression is as follows:

[0034]

[0035] In the formula, l m represents the brightness comparison of the jth scale, V N represents the weight coefficient of brightness comparison of the jth scale, n represents the number of the highest scale, c j represents the contrast comparison of the jth scale, ρ j represents the weight coefficient of contrast comparison of the jth scale, s j represents the structural comparison of the jth scale, μ j represents the weight coefficient of the structural comparison of the jth scale;

[0036] Discriminator loss function The expression is:

[0037]

[0038] Where F2 represents the real high spatial resolution remote sensing image at time t2, and D(F2, C2) represents the identification result obtained after the predicted remote sensing image C2 at time t2 and the real high spatial resolution remote sensing image F2 at time t2 are input into the discriminator.

[0039] In one embodiment, the image reconstruction network includes an upsampling residual module, which upsamples the intermediate features through transposed convolution plus skip connection to obtain a spatiotemporal fused remote sensing image at time t2.

[0040] In a second aspect, the present application provides a remote sensing image spatiotemporal fusion device based on a generative adversarial network, the device comprising:

[0041] The remote sensing image acquisition module is used to acquire a reference remote sensing image F1 at time t1 and a predicted remote sensing image C2 at time t2; the reference remote sensing image at time t1 is a remote sensing image with low temporal resolution and high spatial resolution, and the predicted remote sensing image at time t2 is a remote sensing image with high temporal resolution and low spatial resolution;

[0042] The image feature extraction module is used to extract the spatial features of different scales of the reference remote sensing image F1 at time t1 and the temporal features of different scales of the predicted remote sensing image C2 at time t2 using a switchable multi-scale convolutional attention network to obtain a set of spatial features. and a set of temporal features

[0043] Image feature fusion module is used to integrate spatial features and time characteristics Feature fusion is performed through a dual parallel attention feature fusion module to obtain intermediate features for image reconstruction. The dual parallel attention feature fusion module includes a first attention module and a second attention module. The first attention module is a parallel fusion of global attention and local attention, and the second attention module is a parallel fusion of channel attention and spatial attention. The first attention module is based on the spatial features. and time characteristics After obtaining the preliminary fusion features, the second attention module is based on the spatial features Time characteristics And the initial fusion features get the intermediate features

[0044] The spatiotemporal fusion remote sensing image generation module converts the intermediate features Input into the image reconstruction network to obtain the spatiotemporal fusion remote sensing image at time t2

[0045] The above-mentioned remote sensing image spatiotemporal fusion method and device based on generative adversarial network introduces a switchable multi-scale convolutional attention network and a dual parallel attention feature fusion module in the generator of the generative adversarial network. The switchable multi-scale convolutional attention network can adaptively adjust the feature extraction at different scales, so that the model can effectively capture key information even with limited reference images, significantly reducing the requirement for the number of reference images; at the same time, the dual parallel attention feature fusion module includes a first attention module that is a parallel fusion of global attention and local attention and a second attention module that is a parallel fusion of channel attention and spatial attention. The first attention module generates preliminary fusion features by capturing global trends and local details in the image, thereby enhancing the richness and accuracy of feature representation. The second attention module further utilizes the spatial features, temporal features and preliminary fusion features of the remote sensing image, and utilizes the parallel reinforcement of channel and spatial attention to deeply explore the intrinsic connections and importance differences between features, thereby generating more refined and discriminative intermediate features. Therefore, this method reduces the dependence on the number of reference images and improves the efficiency and accuracy of spatiotemporal fusion of remote sensing images by optimizing the image feature extraction process using a switchable multi-scale convolutional attention network and optimizing the feature fusion process using a dual-parallel attention feature fusion module. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a flow chart of a remote sensing image spatiotemporal fusion method based on a generative adversarial network in one embodiment;

[0047] Figure 2 Schematic diagram of the structure of a down-sampling multi-scale convolutional attention block in one embodiment;

[0048] Figure 3 is a schematic diagram of the structure of a dual parallel attention feature fusion module in one embodiment;

[0049] Figure 4 A schematic diagram of a structure of a generative adversarial network in one embodiment;

[0050] Figure 5 A schematic diagram of detection results of different models on the CIA dataset in one embodiment;

[0051] Figure 6 It is a schematic diagram of detection results of different models on the LGC dataset in one embodiment;

[0052] Figure 7 It is a heat scatter diagram of the detection results of different models on the CIA dataset in one embodiment;

[0053] Figure 8 It is a heat scatter diagram of the detection results of different models on the LGC data set in one embodiment;

[0054] Fig. 9 It is a structural block diagram of a remote sensing image spatiotemporal fusion device based on a generative adversarial network in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] Since the introduction of Generative Adversarial Networks (GANs), many GAN-based image processing models have emerged, playing a key role in the field of image modeling and processing. The core architecture of GANs includes a generator and a discriminator: the generator learns from random noise to generate samples that are very similar to real data, while the discriminator distinguishes whether the data comes from the real data set or the product of the generator. These two parts promote each other through iterative training to achieve the purpose of high-fidelity generated samples and high-precision discrimination. The remote sensing image spatiotemporal fusion method of this application performs spatiotemporal fusion of remote sensing images based on the improved generative adversarial network. The generator in the improved generative adversarial network includes a switchable multi-scale convolutional attention network (SMSCAN) and a dual parallel attention feature fusion module (DPAFF).

[0057] In one embodiment, Figure 1 As shown, a remote sensing image spatiotemporal fusion method based on a generative adversarial network is provided. This embodiment takes the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0058] Step S1, obtaining a reference remote sensing image F1 at time t1 and a predicted remote sensing image C2 at time t2.

[0059] Among them, the reference remote sensing image at time t1 is a remote sensing image with low temporal resolution and high spatial resolution, and the predicted remote sensing image at time t2 is a remote sensing image with high temporal resolution and low spatial resolution.

[0060] Step S2, a switchable multi-scale convolutional attention network is used to extract spatial features of different scales of the reference remote sensing image F1 at time t1 and temporal features of different scales of the predicted remote sensing image C2 at time t2, and obtain a set of spatial features and a set of temporal features

[0061] The switchable multi-scale convolutional attention network includes two branches, one for extracting spatial features of different scales of the reference remote sensing image F1 at time t1, and the other for extracting temporal features of different scales of the predicted remote sensing image C2 at time t2. Each branch includes multiple consecutive down-sampled multi-scale convolutional attention blocks to extract multi-scale spatiotemporal features from the image. Spatial features Time characteristics represents the multi-scale spatial features of the i-th layer, represents the multi-scale temporal features of the i-th layer, and N is the number of layers of the extracted multi-level features.

[0062] Specifically, each downsampled multi-scale convolutional attention block performs convolution operations through convolution kernels of different scales to extract features of different scales and capture contextual information of different ranges, so as to identify and process different scale features in the image from local details to global structures. In the process of extracting features, the downsampled multi-scale convolutional attention block strengthens key features through weighted fusion, while relatively weakening redundant or unimportant information. In addition, the downsampled multi-scale convolutional attention block can analyze the importance of each scale feature based on the relevance and importance of the features, and dynamically adjust the weights of each scale feature according to the analysis results, so that more significant and valuable features receive more attention. This adaptive capability enables the network to automatically focus on key information in the image.

[0063] At the same time, a switchable normalization layer (SN) is added to the downsampled multi-scale convolutional attention block to replace the traditional BatchNorm. Switchable normalization is a highly flexible and data-driven adaptive normalization strategy. Its core advantage is that it can learn and dynamically adjust the key normalization parameters in the network to adapt to the characteristics of different tasks and datasets. Specifically, the switchable normalization layer optimizes the normalization process by simultaneously considering the statistical information (i.e., mean and variance) of three dimensions: channel, layer, and mini-batch. This multi-dimensional consideration enables the switchable normalization layer to maintain excellent performance stability under various batch sizes, and even in small batch scenarios, it can show strong robustness. The design of the switchable normalization layer abandons the reliance on sensitive hyperparameters, simplifies the network configuration process, and maintains the lightweight characteristics of the overall architecture. This not only reduces the difficulty of implementation, but also promotes its rapid deployment and efficient application in various application scenarios.

[0064] Step S3: transform the spatial features and time characteristics Feature fusion is performed through a dual parallel attention feature fusion module to obtain intermediate features for image reconstruction.

[0065] Although the traditional attention mechanism can guide the network to assign different feature weights according to the importance of the features. However, it is impossible to fully fuse the high- and low-resolution multi-scale features obtained in the feature extraction stage. Remote sensing images usually have six channels and large spatial variations, so it is very important to guide the network to learn channel and spatial feature information. This application introduces a dual-parallel attention feature fusion mechanism for feature fusion, replacing the ordinary channel cascade method to fully fuse the extracted features containing multi-scale contextual information. The dual-parallel attention feature fusion module gradually adjusts the weights of the feature map in multiple iterations, allowing the network to correct and optimize the feature representation multiple times until the best fusion effect is achieved. By optimizing the feature fusion process, the dual-parallel attention feature fusion module can better capture the detailed features of the target object, thereby improving the fusion accuracy.

[0066] The dual parallel attention feature fusion module includes the first attention module and the second attention module. The first attention module is a parallel fusion of global attention and local attention, allowing it to focus on the global features and local details of remote sensing images at the same time. The second attention module is a parallel fusion of channel attention and spatial attention. After fully integrating the multi-scale features at the global and local levels, the channel attention and spatial attention are connected in parallel to enhance the fusion effect from the perspective of channel and space.

[0067] Specifically, the first attention module is based on the spatial features and time characteristics After obtaining the preliminary fusion features, the second attention module is based on the spatial features Time characteristics And the preliminary fusion features are used to obtain the intermediate features for image reconstruction The specific feature fusion process is shown in formula (1) and formula (2):

[0068]

[0069]

[0070] In the formula, It represents the spatiotemporal fusion of the multi-scale spatial features of the i-th layer and the multi-scale temporal features of the i-th layer. represents the local attention weight of the fused features, represents the global attention weight of the fused feature, z represents the initial fused feature obtained through the global and local attention parallel structure, M C (z) represents the channel attention weight of the initial fusion feature, M S (z) represents the spatial attention weight of the initial fusion feature, It means that the dual parallel attention feature fusion module fuses the multi-scale features of high and low resolution remote sensing images extracted by global, local, channel and spatial attention, that is, the intermediate features used for image reconstruction. Among them, the local attention weight, global attention weight, channel attention weight and spatial attention weight are real numbers ranging from 0 to 1. Figure 3 The left dashed line in the equation (1) represents Calculation, the dashed line on the right represents (1-(M C (z)+M S (z))) calculation.

[0071] Step S4: The intermediate features Input into the image reconstruction network to obtain the spatiotemporal fusion remote sensing image at time t2

[0072] The image reconstruction network can include multiple convolutional layers, deconvolutional layers, upsampling layers, downsampling layers, and possible residual connections. Through a series of nonlinear transformations and mappings, the spatiotemporal fusion remote sensing image at time t2 is generated. To achieve the reconstruction of image content.

[0073] In addition, during the training phase of the generative adversarial network, the spatiotemporal fusion remote sensing image at time t2 needs to be further The input is sent to the discriminator, which determines whether the input image is a real observed image or a fused image. The discriminator uses a simple CNN structure, and each block consists of multiple convolutional layers, spectral normalization functions (SpectralNorm) and leaky rectified linear unit (LeakyReLU) activation functions. The spectral normalization function can stabilize the adversarial process, and the convolution operation maintains the size of the receptive field of 70×70.

[0074] Specifically, when the input of the discriminator is the reference remote sensing image F2 at time t2 and the predicted remote sensing image C2 at time t2, the discriminator is expected to produce a true matrix, that is, to determine that the input image is a real observed image. On the contrary, when the input of the discriminator is the spatiotemporal fusion remote sensing image at time t2 When the predicted remote sensing image C2 at time t2 is calculated, the discriminator is expected to produce a completely false matrix, that is, to determine that the input image is a fused image. The adversarial signal provided by the discriminator (that is, its output matrix) is used as part of the generator's loss function to encourage the generator to generate images that are more difficult to be distinguished by the discriminator, that is, higher quality fused images. By alternately training the generator and the discriminator, the game process between the two gradually reaches a balance point, at which point the generator can generate a fused image that is close to the real one.

[0075] In the spatiotemporal fusion method of remote sensing images based on a generative adversarial network in this embodiment, a switchable multi-scale convolutional attention network and a dual-parallel attention feature fusion module are introduced in the generator of the generative adversarial network. The switchable multi-scale convolutional attention network can adaptively adjust the feature extraction at different scales, so that the model can effectively capture key information even with limited reference images, significantly reducing the requirement for the number of reference images; at the same time, the dual-parallel attention feature fusion module includes a first attention module that is a parallel fusion of global attention and local attention and a second attention module that is a parallel fusion of channel attention and spatial attention. The first attention module generates preliminary fusion features by capturing global trends and local details in the image, thereby enhancing the richness and accuracy of feature representation. The second attention module further utilizes the spatial features, temporal features and preliminary fusion features of the remote sensing image, and utilizes the parallel reinforcement of channel and spatial attention to deeply explore the intrinsic connections and importance differences between features, thereby generating more refined and discriminative intermediate features. Therefore, this method reduces the dependence on the number of reference images and improves the efficiency and accuracy of spatiotemporal fusion of remote sensing images by optimizing the image feature extraction process using a switchable multi-scale convolutional attention network and optimizing the feature fusion process using a dual-parallel attention feature fusion module.

[0076] In one embodiment, the switchable multi-scale convolutional attention network includes at least one down-sampled multi-scale convolutional attention block, each of which includes a switchable normalization layer (SwitchNorm), a multi-scale convolutional attention layer (Attention) and a feed-forward network layer (FFN). The multi-scale convolutional attention layer includes depth convolution, multi-branch depth strip convolution and 1×1 convolution.

[0077] Specifically, the downsampled multi-scale convolutional attention block first aggregates local information through ordinary deep convolution, and then uses multi-branch deep striped convolution to capture contextual information of different scales, with the convolution kernel size of each branch being 5, 7, and 11 respectively. In particular, in order to reduce the computational burden, deep striped convolution is used here to simulate the effect of standard two-dimensional convolution, such as a pair of 5×1 and 1×5 convolutions replacing 5×5 convolutions. Then, the contextual information of different scales captured by the multi-branch deep striped convolution is channel-fused with the local information aggregated by the ordinary deep convolution, and the 1×1 convolution is used to adjust the number of channels of the image after channel fusion and model the relationship between channels. This multi-scale design effectively aggregates rich semantic information, reduces the redundancy of the self-attention mechanism, and improves efficiency. The downsampled multi-scale convolutional attention block fully extracts feature context by enhancing the salient areas in the image, which is beneficial to subsequent image reconstruction.

[0078] In the down-sampling multi-scale convolutional attention block, FFN follows the multi-scale convolutional attention layer and plays a role similar to the nonlinear transformation component in the Transformer structure. Specifically, FFN uses a series of convolutional layers, such as Figure 2 The 1×1 convolution and 3×3 convolution in the convolutional layer perform linear mapping in different dimensions and introduce necessary nonlinearity through nonlinear activation functions, such as the LeakyReLU function, to help the model capture high-order interaction characteristics in the data. This structural design enables FFN to increase the expressiveness of the model, that is, the model can describe and distinguish more complex and diverse data patterns. At the same time, the introduction of FFN also enhances the learning potential of the model, enabling it to show higher adaptability and generalization capabilities when facing complex tasks.

[0079] The multi-branch deep strip convolution in this embodiment can screen and emphasize important features, while the FFN network enhances the expressiveness of these features by introducing nonlinearity. Together, they promote a deep understanding of the input data and high-quality feature representation, reduce dependence on the number of reference images, and improve the reliability of feature fusion.

[0080] In one embodiment, Figure 3 As shown, the local attention consists of two 1×1 convolutional layers, and the global attention consists of a global average pooling layer and two 1×1 convolutional layers.

[0081] Among them, the input of the first 1×1 convolutional layer of local attention is the spatial feature With time characteristics The output features of the first 1×1 convolutional layer are batch normalized (BN) and ReLU activated before being input into the second 1×1 convolutional layer. The output of the second 1×1 convolutional layer is then batch normalized again.

[0082] Specifically, the first 1×1 convolutional layer extracts preliminary feature representations by changing the depth (i.e., the number of channels) of the feature map while keeping the spatial dimension of the input feature map unchanged. Figure 3 As shown in the figure C in the formula represents the number of channels of the input feature map, and r represents the ratio of channel compression, which is usually an integer greater than 1. It means that after the first 1×1 convolution layer, the number of channels of the input feature map is reduced to the original The purpose of channel compression is to reduce the amount of calculation and parameters. The second 1×1 convolutional layer is used to expand the number of channels of the input feature map to the original number of channels C.

[0083] The input of the global average pooling layer is the spatial feature With time characteristics The fused feature map. Global attention first averages the input feature map in the spatial dimension through the global average pooling layer to obtain an output feature map (C×1×1) with the same number of channels as the input feature map but a spatial dimension of 1×1. This process effectively compresses the information of each channel into a global representation that reflects the average activation level of the channel over the entire image. Next, the output feature map of the global average pooling layer is input into the first 1×1 convolutional layer. The first 1×1 convolutional layer extracts a preliminary feature representation by changing the depth (i.e., the number of channels) of the feature map while keeping the spatial dimension of the input feature map unchanged, and obtains the output feature map. The output feature map of the first 1×1 convolutional layer is batch normalized and ReLU activated before being input to the second 1×1 convolutional layer, which restores the number of channels of the input feature map to C. The output feature map (C×1×1) of the second 1×1 convolutional layer is then batch normalized.

[0084] Finally, after obtaining the feature representations of local attention and global attention, the features after global attention and local attention are combined with the spatial features by adding feature maps element by element. and time characteristics Combined together, we get the preliminary fusion features, which contain both local details of the image and global context information, providing rich feature representation.

[0085] The local attention of this embodiment includes two 1×1 convolutional layers. By stacking two 1×1 convolutional layers, the local attention module can focus on the local area in the image and extract rich detail features; the global attention includes a global average pooling layer and two 1×1 convolutional layers. The global average pooling layer compresses the feature map in the spatial dimension and extracts the global feature representation. These global features are then transformed and enhanced through two 1×1 convolutional layers, which helps the model to grasp the global context of the image, thereby being more robust when processing complex scenes and improving the reliability of spatiotemporal fusion of remote sensing images.

[0086] The goal of spatial attention is to emphasize important spatial locations in the input feature map while suppressing less important areas. This helps the model focus more on the key parts of the object and improves the accuracy of recognition and classification.

[0087] In one embodiment, the spatial attention includes two 1×1 convolutional layers and two dilated convolutional layers.

[0088] The input of the first 1×1 convolutional layer is the preliminary fusion feature obtained through local attention and global attention. The main function of the first 1×1 convolutional layer is to compress the number of channels of the input feature map, reducing it to 1 / r of the original number of channels to reduce the amount of calculation while retaining enough spatial information for subsequent processing. The output feature map of the first 1×1 convolutional layer is input to the first dilated convolutional layer (Dialition_Conv3) after batch normalization and ReLU activation. Output feature map of the first dilated convolutional layer After batch normalization and ReLU activation, it is input to the second dilated convolution layer (Dialition_Conv3). Similarly, the output feature map of the second dilated convolution layer is After batch normalization and ReLU activation, it is input to the second 1×1 convolution layer. The second 1×1 convolution layer changes the number of channels to 1 and generates a spatial attention map (1×H×W). The spatial attention map is a single-channel map with the same spatial size as the initial fusion feature. Each value in the spatial attention map represents the importance weight of the corresponding position.

[0089] The spatial attention in this embodiment firstly performs efficient channel compression on the input feature map through the first 1×1 convolution layer, reducing the amount of calculation of subsequent layers, and then increases the effective receptive field of the convolution kernel through two dilated convolution layers, so that the model can capture a wider range of contextual information and long-distance spatial dependencies without adding additional parameters. This spatial attention not only enhances the model's ability to understand and express complex remote sensing image features, but also significantly improves the reliability and efficiency of spatiotemporal fusion tasks.

[0090] The dilated convolution layer increases the effective receptive field of the convolution kernel by introducing a dilation rate, while keeping the number of parameters and the amount of computation relatively unchanged. The dilation rate determines the sampling interval of the convolution kernel on the input data, that is, the distance between adjacent elements.

[0091] In one embodiment, the dilated convolution layer has a dilation rate of 4. When the dilation rate is set to 4, it means that a 3×3 zero-value region is formed around each element of the convolution kernel. At this time, the actual sampling points of the convolution kernel on the input data become sparser, but each sampling can cover farther input data points, thereby significantly increasing the receptive field without increasing the physical size of the convolution kernel.

[0092] In one embodiment, Figure 3 As shown, the channel attention includes an average pooling branch and a maximum pooling branch, the average pooling branch is connected in parallel with the maximum pooling branch, the average pooling branch includes an average pooling layer and a first fully connected layer, and the maximum pooling branch includes a maximum pooling layer and a second fully connected layer.

[0093] Among them, the channel attention includes two parallel processing branches, which use the maximum pooling method and the average pooling method to obtain the global statistical features of the preliminary fusion features. For the average pooling branch, the preliminary fusion features are subjected to global average pooling through the average pooling layer. Global average pooling calculates the average value for each channel of the feature map, thereby obtaining a feature vector with the same number of channels as the preliminary fusion features but a spatial dimension (height and width) of 1. This feature vector reflects the average feature strength of each channel. For the maximum pooling branch, the preliminary fusion features are subjected to global maximum pooling through the maximum pooling layer. Similar to average pooling, global maximum pooling also operates on each channel separately, but it calculates the maximum value of each channel and also obtains a feature vector with a spatial dimension of 1, which reflects the most significant feature of each channel. After obtaining the global statistical features, both branches will pass through one or more fully connected layers (FC) for feature transformation and dependency modeling. For example Figure 3In the above example, the two branches generate their own channel features through two fully connected layers, and then the channel features of the two branches are added element by element to generate the final channel attention weight. Each element in the channel attention weight can be regarded as the optimal weight of the corresponding feature channel.

[0094] Further, such as Figure 3 As shown in FIG. 1 , the output of the fully connected layer can also be batch normalized to accelerate the training process and reduce overfitting, thereby improving the efficiency of spatiotemporal fusion of remote sensing images. In addition, the ReLU activation function can be applied to the output of the fully connected layer after batch normalization to introduce nonlinearity, thereby enhancing the expressive power of the model and improving the reliability of spatiotemporal fusion of remote sensing images.

[0095] The channel attention in this embodiment includes two parallel processed maximum pooling branches and average pooling branches. By using maximum pooling and average pooling in parallel to obtain the global statistical information of the input feature map, the model can simultaneously consider the most significant features and average features in the feature channel, so that the model can be more comprehensive and accurate in judging the importance of feature channels, further improving the reliability of spatiotemporal fusion of remote sensing images.

[0096] In one embodiment, the remote sensing image spatiotemporal fusion method further includes:

[0097] Step S51: fusion of the remote sensing image at time t2 Input into the discriminator for identification and obtain the identification result.

[0098] Among them, the spatiotemporal fusion remote sensing image at time t2 is the spatiotemporal fusion remote sensing image at time t2 generated by the generator in the conditional generative adversarial network. The task of the discriminator is to distinguish whether the input image is the real remote sensing image F2 at time t2 or the spatiotemporal fusion remote sensing image generated by the generator. Through training, the discriminator can learn the difference features between the real remote sensing image and the fused image, and thus give the identification result.

[0099] Step S52, based on the identification result, the reference remote sensing image F1 at time t1, the predicted remote sensing image C2 at time t2 and the spatiotemporal fusion remote sensing image at time t2 Calculate the value of the loss function.

[0100] Among them, the loss function can include pixel-level loss (such as L1 or L2 loss), adversarial loss and structural loss between the generated image and the real image. Pixel-level loss directly compares the values ​​of the two images at corresponding pixel positions and calculates the difference between them. Adversarial loss is used to encourage the generated image to deceive the discriminator, usually by maximizing the probability of the discriminator misclassifying the generated image. Structural loss is used to ensure that the generated fusion image retains important features while maintaining a similar structure to the original image.

[0101] Step S53, updating the parameters of the switchable multi-scale convolutional attention network, the dual parallel attention feature fusion module and the discriminator according to the value of the loss function.

[0102] Specifically, according to the calculated loss function value, the back propagation algorithm is used to update the parameters of the switchable multi-scale convolutional attention network, the dual parallel attention feature fusion module and the discriminator. This process is iterative and aims to minimize the loss function so that the generated spatiotemporal fusion remote sensing image at time t2 is visually closer to the real remote sensing image at time t2.

[0103] Step S54, repeat steps S51 to S53 until the value of the loss function meets the preset conditions, and use the parameters of the current switchable multi-scale convolutional attention network, dual parallel attention feature fusion module and discriminator as the final model parameters.

[0104] The preset conditions may include the loss function value falling below a certain threshold, the loss value no longer decreasing significantly, or the number of iterations reaching an upper limit. When the loss function value meets the preset conditions, the model is considered to have converged, and the parameters of the current switchable multi-scale convolutional attention network, dual parallel attention feature fusion module, and discriminator are used as the final model parameters. These parameters will be used to generate high-quality spatiotemporal fusion remote sensing images.

[0105] This embodiment can further improve the efficiency and reliability of spatiotemporal fusion of remote sensing images by iteratively training the generative adversarial network and optimizing model parameters.

[0106] In one embodiment, the loss function includes a conditional generative network total loss function and a discriminator loss function. The expression of is shown in formula (3):

[0107]

[0108] In the formula, represents the least squares generative adversarial network loss function, represents the L1 loss function, which is used to calculate the sum of the absolute values ​​of the differences between the corresponding pixels of the generated image and the real image. It is less sensitive to outliers, that is, points far from the average value, and helps to generate clearer image edges. represents the spectral loss function, represents the structural loss function, α represents the weight of the least squares generative adversarial network loss function, and β represents The weight of the loss function, γ represents the weight of the spectral loss function, and δ represents the weight of the structural loss function.

[0109] Least Squares Generative Adversarial Network Loss Function The specific expression of is shown in formula (4):

[0110]

[0111] In the formula, F1 represents the reference remote sensing image at time t1, C2 represents the predicted remote sensing image at time t2, E represents the expectation, and G(F1, C2) represents the spatiotemporal fusion remote sensing image at time t2 obtained by inputting the reference remote sensing image F1 at time t1 and the predicted remote sensing image C2 at time t2 into the generator. D(G(F1, C2), C2) represents the spatiotemporal fusion of the predicted remote sensing image C2 at time t2 and the remote sensing image C2 at time t2. The identification result obtained after input into the discriminator.

[0112] The specific expression of the loss function is shown in formula (5):

[0113]

[0114] Where K represents the number of pixels of the remote sensing image, T represents the real image matrix, P represents the predicted image matrix, and ||.||1 represents the L1 norm.

[0115] Spectral loss function The specific expression of is shown in formula (6):

[0116]

[0117] Where I is a tensor of all 1s, and ||.||2 represents the L2 norm.

[0118] Structural loss function The specific expression of is shown in formula (7):

[0119]

[0120] In the formula, l m represents the brightness comparison of the jth scale, V Nrepresents the weight coefficient of brightness comparison of the jth scale, n represents the number of the highest scale, c j represents the contrast comparison of the jth scale, ρ j represents the weight coefficient of contrast comparison of the jth scale, s j represents the structural comparison of the jth scale, μ j Represents the weight coefficient of the structural comparison of the j-th scale.

[0121] Discriminator loss function The specific expression of is shown in formula (8):

[0122]

[0123] Wherein, F2 represents the real high spatial resolution remote sensing image at time t2, and D(F2, C2) represents the identification result obtained after the predicted remote sensing image C2 at time t2 and the real high spatial resolution remote sensing image F2 at time t2 are input into the discriminator.

[0124] In addition to the loss function of the traditional least squares generative adversarial network, the loss function of the generator in this embodiment also designs an L1 loss function, a spectral loss function, and a structural loss function to ensure the authenticity and accuracy of the generated image. The L1 loss function ensures the consistency of the generated image with the real image at the pixel level, while the spectral loss function and the structural loss function focus on the accurate capture of spectral similarity and multi-scale structural similarity, respectively, further improving the detail richness and overall quality of image generation. In addition, by giving flexible and adjustable weights to the four loss functions, the model can be made more flexible, thereby meeting the specific needs and standards of spatiotemporal fusion of remote sensing images in different application scenarios.

[0125] In one embodiment, the image reconstruction network includes an upsampling residual module, which upsamples the intermediate features through transposed convolution plus skip connection to obtain a spatiotemporal fused remote sensing image at time t2.

[0126] Among them, the image reconstruction network is used to convert the processed feature information into a high-quality spatiotemporal fused remote sensing image. The input of the image reconstruction stage is the features processed by the dual parallel attention feature fusion module. During the reconstruction process, each upsampled residual block accepts the feature input from the previous layer and upsamples it through transposed convolution. Then, it combines the multi-level features containing multi-scale contextual information in the feature extraction stage introduced by the jump connection to generate a feature map with larger size and richer details. The final generated spatiotemporal fused remote sensing image at time t2 It is obtained by layer-by-layer construction and fusion of multi-layer upsampling residual blocks. It inherits feature information from different levels and can therefore present higher spatial resolution and richer detail information.

[0127] The image reconstruction network in this embodiment increases the size of the feature map by transposed convolution, and combines jump connections with multi-level features containing multi-scale context information in the feature extraction stage, which can improve the accuracy and reliability of image reconstruction.

[0128] In order to better understand the remote sensing image spatiotemporal fusion method based on the generative adversarial network in the above embodiment, a more detailed embodiment is provided below for explanation.

[0129] In one embodiment, the specific structure of the generative adversarial network in the remote sensing image spatiotemporal fusion method based on the generative adversarial network is as follows: Figure 4 As shown, it includes a generator and a discriminator. The generator includes an image feature extraction network, an image feature fusion network and an image reconstruction network. This method adds a multi-scale convolutional attention mechanism and a dual parallel attention feature fusion mechanism to the image feature extraction network, and introduces residual connections in the two processes of feature extraction and image reconstruction, retaining the multi-level and multi-scale context features in the feature extraction process. The use of jump connections for image reconstruction can effectively avoid gradient vanishing and gradient explosion, and compensate for the loss of spatial resolution. This method only requires one remote sensing image with high temporal resolution and low spatial resolution and one remote sensing image with low temporal resolution and high spatial resolution to generate a fused image with good effect, which solves the problem that the existing methods require a large number of reference images.

[0130] The image feature extraction network includes the F1 image feature extraction network and the C2 image feature extraction network. The F1 image feature extraction network is used to extract spatial features of different scales of the reference remote sensing image F1 at time t1. The C2 image feature extraction network is used to extract temporal features of different scales of the predicted remote sensing image C2 at time t2. Both the F1 image feature extraction network and the C2 image feature extraction network include a cascaded 1×1 convolution layer and four downsampled multi-scale convolutional attention blocks. Each downsampled multi-scale convolutional attention block includes a switchable multi-scale convolutional attention network (SMSCAN), a 3×3 convolutional layer, and a ReLU activation layer. The specific structure of the downsampled multi-scale convolutional attention block is as follows: Figure 2 The image feature fusion network includes a dual parallel attention feature fusion module (DPAFF), the specific structure is as follows Figure 3 shown.

[0131] The image reconstruction network consists of four 3×3 transposed convolutional layers and one 1×1 convolutional layer connected in cascade, and the ReLU activation function is applied to the output of each transposed convolutional layer and convolutional layer. The input of the first 3×3 transposed convolutional layer includes the intermediate features output by the dual parallel attention feature fusion module. and the output features of the 1×1 convolutional layer in the F1 image feature extraction network, the input of the second 3×3 transposed convolutional layer includes the output of the first 3×3 transposed convolutional layer and the output features of the first feature extraction module in the F1 image feature extraction network, the input of the third 3×3 transposed convolutional layer includes the output of the second 3×3 transposed convolutional layer and the output features of the second feature extraction module in the F1 image feature extraction network, the input of the fourth 3×3 transposed convolutional layer includes the output of the third 3×3 transposed convolutional layer and the output features of the third feature extraction module in the F1 image feature extraction network, and the input of the 1×1 convolutional layer includes the output of the fourth 3×3 transposed convolutional layer and the output features of the fourth feature extraction module in the F1 image feature extraction network.

[0132] The discriminator uses a simple CNN structure. Each block of the discriminator consists of multiple convolutional layers, spectral normalization (SpectralNorm) and leaky rectified linear unit (LeakyReLU) activation functions. Spectral normalization can stabilize the adversarial process, while the convolution operation maintains the size of the receptive field of 70×70. During the training phase of the discriminator, when its input is F2 and C2, it is expected to produce a true matrix. On the contrary, when its input is When C2 is used, it is expected to produce a completely fake matrix. The adversarial function provided by the discriminator helps the generator obtain higher quality results. Specifically, Figure 4 As shown, the discriminator consists of a 1×1 convolutional layer, four spectral normalization modules, another 1×1 convolutional layer, and a PatchMap layer connected in cascade.

[0133] In order to verify the effectiveness of this method, the generative adversarial network in this method is used as the remote sensing image spatiotemporal fusion model of this embodiment, and relevant experiments are carried out on two public remote sensing datasets CIA and LGC, as follows:

[0134] Quantitative experiments:

[0135] This embodiment selects four models: STARFM (Spatial and Temporal Adaptive Reflectance Fusion Model), FSDAF (Flexible Spatiotemporal Data Fusion), DCSTFN (Deep Convolutional Spatial-Temporal Fusion Network) and GANSTFM (Generative Adversarial Network-based Spatiotemporal Fusion Model), and conducts comparative experiments with the remote sensing image spatiotemporal fusion model of this embodiment on the public remote sensing datasets CIA dataset and LGC dataset. STARFM and FSDAF are traditional methods with good performance, while DCSTFN and GANSTFM are classic methods based on deep learning. DCSTFN adopts the CNN framework, and GANSTFM adopts the GAN architecture. The specific quantitative analysis of the experimental results of the CIA dataset is shown in Table 1.

[0136] Table 1 Experimental results related to CIA dataset

[0137]

[0138]

[0139] The comparative experiment comprehensively evaluates the effect of the model from six commonly used evaluation indicators, namely, peak signal-to-noise ratio (PSNR), structural similarity (SSIM), spectral angle mapper (SAM), relative dimensionless global synthesis error (ERGAS), spatial correlation coefficient (CC) and root mean square error (RMSE). The higher the SSIM, PSNR and CC values, the higher the similarity between the image obtained by spatiotemporal fusion and the real image, and the smaller the RMSE, SAM and ERGAS values, the better the spatiotemporal fusion effect and the stability of the model. The experimental results show the experimental effects and average values ​​of the six bands of remote sensing images on SSIM, PSNR, CC, RMSE, as well as the experimental results on SAM and ERGAS. The final results show that the spatiotemporal fusion model of remote sensing images proposed in this embodiment has reached the optimal value in almost all indicators. For example, in the CIA data set, in terms of the SSIM index related to the image structure similarity, the results of the spatiotemporal fusion model of this embodiment are about 10.1% and 1.9% higher than those of the traditional STARFM and FSDAF methods, respectively. Compared with the DCSTFN and GANSTFM methods based on deep learning, the spatiotemporal fusion model of this embodiment improves SSIM by about 1.2% and 1.5%, respectively. In the LGC dataset, the results of the spatiotemporal fusion model of this embodiment are about 10.8% and 4.2% higher than the traditional STARFM and FSDAF methods, respectively. Compared with the DCSTFN and GANSTFM methods based on deep learning, the spatiotemporal fusion model of this embodiment improves by about 3.3% and 1.6%, respectively. We attribute these improvements to several factors. First, the powerful model performance of the generative adversarial network produces clearer and more realistic samples. Secondly, the spatiotemporal fusion model of remote sensing images in this embodiment combines multi-scale convolutional attention (SMSCAN), which can capture detailed features of multiple scales while effectively processing spatial and distance dependencies in the image, and at the same time, it allocates attention weights based on multiple scales, and can pay more attention to more important multi-scale features. In the feature fusion stage, the dual parallel attention feature fusion module (DPAFF) is used to fully fuse the extracted multi-scale features. Different attention weights are assigned according to the contribution of the multi-scale features obtained in the feature extraction stage to the final output, achieving excellent fusion effect.

[0140] The specific quantitative analysis of the experimental results of the LGC dataset is shown in Table 2.

[0141] Table 2 Experimental results related to LGC dataset

[0142]

[0143]

[0144] As shown in Table 2, the spatiotemporal fusion model of this embodiment is significantly better than other models in terms of PSNR and SSIM, which indicates that the spatiotemporal fusion model of this embodiment extracts structural information and texture details more effectively. Figure 8 A heat map of the prediction results of each model in the LGC dataset is compared. The figure shows the high "point cloud" density of all models, which is a feature of the LGC dataset. However, the greater the model density, the higher the image resolution in the reconstruction stage, which may enhance the target detection ability of the final remote sensing image. Compared with other models, the scatter points of the spatiotemporal fusion model of this embodiment are densely concentrated near the diagonal. Overall, the scatter point distribution of the spatiotemporal fusion model of this embodiment in each band is closely aligned with the diagonal, and the bright areas are concentrated near the diagonal, indicating that the prediction results of the spatiotemporal fusion model of this embodiment are closer to the actual observations. This also proves the robustness of the spatiotemporal fusion method of remote sensing images in this embodiment when dealing with complex changes.

[0145] Qualitative Experiments

[0146] In order to intuitively represent our experimental results, Figure 5 The quantitative experimental results of STARFM, FSDAF, DCSTFN, GANSTFM and the remote sensing image spatiotemporal fusion model (OURS) of this embodiment on the CIA dataset and LGC dataset are shown. The CIA dataset mainly contains crop phenology data, including many farmland planting and irrigation areas. The color image composed of bands 3, 4, and 5 is selected for visual demonstration. Among them, Figure 5 The first row is an overview of the spatiotemporal fusion images generated by each model. “(a) Observed” is the actual observed image, “(b) STARFM” is the spatiotemporal fusion image generated by STARFM, “(c) FSDAF” is the spatiotemporal fusion image generated by FSDAF, “(d) DCSTFN” is the spatiotemporal fusion image generated by DCSTFN, “(e) GANSTFM” is the spatiotemporal fusion image generated by GANSTFM, and “(f) OURS” is the spatiotemporal fusion image generated by the remote sensing image spatiotemporal fusion model in this embodiment. Figure 5The second row is an enlarged representation of the image within the yellow frame range in the overall effect diagram of the first row (Zoomed Details). It can be seen from the overall effect diagram that the clarity of the spatiotemporal fusion image generated by the remote sensing image spatiotemporal fusion model in this embodiment is closer to the original image. STARFM and FSDAF perform poorly in feature-dense areas, mainly because they focus on spatiotemporal features and ignore the details of texture features. Methods such as DCSTFN and GANSTFM based on deep learning can not only predict dense areas, but also clearly identify the differences between different color areas, and accurately predict information such as farmland and roads. However, the results show that the remote sensing image spatiotemporal fusion model in this embodiment performs better in irregular areas. In addition, Figure 7 The heat scatter plot of the prediction results of each model in the CIA data set in the same band is shown. The horizontal axis of the heat scatter plot is the band of the predicted image, and the vertical axis is the band of the real image. The heat scatter plot is mainly used for positioning and counting targets. It can be seen that compared with other models, the remote sensing image spatiotemporal fusion model in this embodiment has very dense scatter points near the diagonal, and the bright areas are concentrated near the diagonal. This shows that the prediction results of the remote sensing image spatiotemporal fusion model in this embodiment are closer to the actual observation value and have stronger robustness when dealing with complex changes.

[0147] Figure 6 The fusion results of each model on the LGC dataset are shown. Figure 6 As shown, the output image of the STARFM model contains too much noise. This problem arises because STARFM, as an earlier spatiotemporal fusion algorithm, lacks the ability to detect spatial features of regional boundaries (such as farmland and roads). The output image of the FSDAF model has large deviations within the region because the algorithm cannot adapt to dynamic features and repeatedly segments unchanged pixels. In contrast, the spatiotemporal fusion image generated by the remote sensing image spatiotemporal fusion model in this embodiment is very close to the actual image with little overall difference. This accuracy is attributed to the inclusion of multi-scale convolutional attention, which captures detailed features across multiple scales while effectively managing spatial dependencies and long-range dependencies in the image.

[0148] The spatiotemporal fusion method of remote sensing images based on generative adversarial networks in this embodiment introduces a multi-scale convolutional attention mechanism in the feature extraction network. Without increasing too much computational effort, the previous multi-scale mechanism does not distinguish the importance of multi-scale features. When extracting multi-scale context information, different attention weights are assigned to features of different scales according to the degree of importance to enhance the significant area. The dual parallel attention feature fusion module is introduced in feature fusion, and its main purpose is to optimize the fusion of multi-scale feature maps. Compared with the ordinary attention mechanism, the dual parallel attention feature fusion module network gradually adjusts the weights of feature maps in multiple parallel attention structures, allowing the network to correct and optimize feature representations multiple times until the best fusion effect is achieved. It focuses on solving the problem of multi-scale feature fusion and is particularly suitable for processing images containing targets of various sizes. By optimizing the feature fusion process, the dual parallel attention feature fusion module can better capture the detailed features of the target object, thereby improving the fusion accuracy. It can adaptively adjust attention according to the size and shape of the target, so that the model has a good response to targets of different scales.

[0149] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0150] Based on the same inventive concept, the embodiment of the present application also provides a remote sensing image spatiotemporal fusion device based on a generative adversarial network for implementing the remote sensing image spatiotemporal fusion method based on a generative adversarial network involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more remote sensing image spatiotemporal fusion devices based on a generative adversarial network provided below can refer to the limitations of the remote sensing image spatiotemporal fusion method based on a generative adversarial network above, and will not be repeated here.

[0151] In one embodiment, Fig. 9 As shown, a remote sensing image spatiotemporal fusion device 900 based on a generative adversarial network is provided, comprising: a remote sensing image acquisition module 901, an image feature extraction module 902, an image feature fusion module 903 and a spatiotemporal fusion remote sensing image generation module 904, wherein:

[0152] The remote sensing image acquisition module 901 is used to acquire a reference remote sensing image F1 at time t1 and a predicted remote sensing image C2 at time t2; the reference remote sensing image at time t1 is a remote sensing image with low temporal resolution and high spatial resolution, and the predicted remote sensing image at time t2 is a remote sensing image with high temporal resolution and low spatial resolution;

[0153] Image feature extraction module 902 is used to extract spatial features of different scales of the reference remote sensing image F1 at time t1 and temporal features of different scales of the predicted remote sensing image C2 at time t2 using a switchable multi-scale convolutional attention network to obtain a set of spatial features and a set of temporal features

[0154] Image feature fusion module 903 is used to combine spatial features and time characteristics Feature fusion is performed through a dual parallel attention feature fusion module to obtain intermediate features for image reconstruction. The dual parallel attention feature fusion module includes a first attention module and a second attention module. The first attention module is a parallel fusion of global attention and local attention, and the second attention module is a parallel fusion of channel attention and spatial attention. The first attention module is based on the spatial features. and time characteristics After obtaining the preliminary fusion features, the second attention module is based on the spatial features Time characteristics And the initial fusion features get the intermediate features

[0155] The spatiotemporal fusion remote sensing image generation module 904 converts the intermediate features Input into the image reconstruction network to obtain the spatiotemporal fusion remote sensing image at time t2

[0156] In one embodiment, the remote sensing image spatiotemporal fusion device 900 based on the generative adversarial network is also used to combine the spatiotemporal fusion remote sensing image at time t2 Input to the discriminator for identification and obtain the identification result; according to the identification result, the reference remote sensing image F1 at time t1, the predicted remote sensing image C2 at time t2 and the spatiotemporal fusion remote sensing image at time t2 Calculate the value of the loss function; update the parameters of the switchable multi-scale convolutional attention network, the dual parallel attention feature fusion module and the discriminator according to the value of the loss function; repeat the above steps until the value of the loss function meets the preset conditions, and use the current parameters of the switchable multi-scale convolutional attention network, the dual parallel attention feature fusion module and the discriminator as the final model parameters.

[0157] Each module in the above-mentioned remote sensing image spatiotemporal fusion device based on generative adversarial network can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0158] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0159] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0160] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0161] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0162] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A remote sensing image spatiotemporal fusion method based on generative adversarial network, characterized in that: The generator in the generative adversarial network includes a switchable multi-scale convolutional attention network and a dual parallel attention feature fusion module, and the method includes: Step S1, obtaining a reference remote sensing image F1 at time t1 and a predicted remote sensing image C2 at time t2; the reference remote sensing image at time t1 is a remote sensing image with low temporal resolution and high spatial resolution, and the predicted remote sensing image at time t2 is a remote sensing image with high temporal resolution and low spatial resolution; Step S2, using the switchable multi-scale convolutional attention network to extract spatial features of different scales of the reference remote sensing image F1 at time t1 and temporal features of different scales of the predicted remote sensing image C2 at time t2, to obtain a set of spatial features and a set of temporal features Step S3: and time characteristics The dual parallel attention feature fusion module is used to perform feature fusion to obtain intermediate features for image reconstruction. The dual parallel attention feature fusion module includes a first attention module and a second attention module, wherein the first attention module is a parallel fusion of global attention and local attention, and the second attention module is a parallel fusion of channel attention and spatial attention; the first attention module is based on the spatial feature and the time characteristics After obtaining the preliminary fusion features, the second attention module uses the spatial features The time characteristics and the preliminary fusion features to obtain the intermediate features Step S4, the intermediate feature Input into the image reconstruction network to obtain the spatiotemporal fusion remote sensing image at time t2 2. The remote sensing image spatiotemporal fusion method based on generative adversarial network according to claim 1, characterized in that: The switchable multi-scale convolutional attention network includes at least one downsampling multi-scale convolutional attention block, each of which includes a switchable normalization layer, a multi-scale convolutional attention layer and a feedforward network layer. The multi-scale convolutional attention layer includes deep convolution, multi-branch deep strip convolution and 1×1 convolution. The convolution kernel size of each branch in the multi-branch deep strip convolution is set to 5, 7 and 11 respectively.

3. The remote sensing image spatiotemporal fusion method based on generative adversarial network according to claim 1, characterized in that: The local attention includes two 1×1 convolutional layers, and the global attention includes a global average pooling layer and two 1×1 convolutional layers.

4. The remote sensing image spatiotemporal fusion method based on generative adversarial network according to claim 1, characterized in that: The spatial attention consists of two 1×1 convolutional layers and two dilated convolutional layers.

5. The remote sensing image spatiotemporal fusion method based on generative adversarial network according to claim 4, characterized in that: The dilated convolutional layer has a hole rate of 4.

6. The remote sensing image spatiotemporal fusion method based on generative adversarial network according to claim 1, characterized in that: The channel attention includes an average pooling branch and a maximum pooling branch, the average pooling branch is connected in parallel with the maximum pooling branch, the average pooling branch includes an average pooling layer and a first fully connected layer, and the maximum pooling branch includes a maximum pooling layer and a second fully connected layer.

7. The remote sensing image spatiotemporal fusion method based on generative adversarial network according to claim 1, characterized in that: The method further comprises: Step S51: The spatiotemporal fusion remote sensing image at time t2 is Input to the discriminator for identification and obtain the identification result; Step S52, based on the identification result, the reference remote sensing image F1 at time t1, the predicted remote sensing image C2 at time t2 and the spatiotemporal fusion remote sensing image at time t2 Calculate the value of the loss function; Step S53, updating the parameters of the switchable multi-scale convolutional attention network, the dual parallel attention feature fusion module and the discriminator according to the value of the loss function; Step S54, repeat steps S51 to S53 until the value of the loss function meets the preset conditions, and use the parameters of the current switchable multi-scale convolutional attention network, the dual parallel attention feature fusion module and the discriminator as the final model parameters.

8. The remote sensing image spatiotemporal fusion method based on generative adversarial network according to claim 7, characterized in that: The loss function includes a conditional generation network total loss function and a discriminator loss function; The condition generates the total loss function of the network The expression is: In the formula, represents the least squares generative adversarial network loss function, represents the L1 loss function, represents the spectral loss function, represents the structural loss function, α represents the weight of the least squares generative adversarial network loss function, β represents the weight of the L1 loss function, γ represents the weight of the spectral loss function, and δ represents the weight of the structural loss function; wherein, The least squares generative adversarial network loss function The expression is: In the formula, F1 represents the reference remote sensing image at time t1, C2 represents the predicted remote sensing image at time t2, E represents the expectation, and G(F1, C2) represents the spatiotemporal fusion remote sensing image at time t2 obtained by inputting the reference remote sensing image F1 at time t1 and the predicted remote sensing image C2 at time t2 into the generator. D(G(F1,C2),C2) represents the spatiotemporal fusion of the predicted remote sensing image C2 at time t2 and the remote sensing image C2 at time t2. The identification result obtained after inputting into the identification device; The L1 loss function The expression is as follows: Where K represents the number of pixels of the remote sensing image, T represents the real image matrix, P represents the predicted image matrix, and ||.||1 represents the L1 norm; The spectral loss function The expression is as follows: Where I is a tensor of all 1s, ||.||2 represents the L2 norm; The structural loss function The expression is as follows: In the formula, l m represents the brightness comparison of the jth scale, V N represents the weight coefficient of brightness comparison of the jth scale, n represents the number of the highest scale, c j represents the contrast comparison of the jth scale, ρ j represents the weight coefficient of contrast comparison of the jth scale, s j represents the structural comparison of the jth scale, μ j represents the weight coefficient of the structural comparison of the jth scale; The discriminator loss function The expression is: In the formula, F2 represents the real high spatial resolution remote sensing image at time t2, and D(F2, C2) represents the identification result obtained after the predicted remote sensing image C2 at time t2 and the real high spatial resolution remote sensing image F2 at time t2 are input into the discriminator.

9. The remote sensing image spatiotemporal fusion method based on generative adversarial network according to claim 1, characterized in that: The image reconstruction network includes an upsampling residual module, which performs upsampling processing on the intermediate features through transposed convolution plus skip connection to obtain the spatiotemporal fusion remote sensing image at time t2.

10. A remote sensing image spatiotemporal fusion device based on a generative adversarial network, characterized in that: The device comprises: A remote sensing image acquisition module, used to acquire a reference remote sensing image F1 at time t1 and a predicted remote sensing image C2 at time t2; the reference remote sensing image at time t1 is a remote sensing image with low temporal resolution and high spatial resolution, and the predicted remote sensing image at time t2 is a remote sensing image with high temporal resolution and low spatial resolution; The image feature extraction module is used to use the switchable multi-scale convolutional attention network to respectively extract the spatial features of different scales of the reference remote sensing image F1 at time t1 and the temporal features of different scales of the predicted remote sensing image C2 at time t2 to obtain a set of spatial features. and a set of temporal features An image feature fusion module is used to combine the spatial features and time characteristics The dual parallel attention feature fusion module is used to perform feature fusion to obtain intermediate features for image reconstruction. The dual parallel attention feature fusion module includes a first attention module and a second attention module, wherein the first attention module is a parallel fusion of global attention and local attention, and the second attention module is a parallel fusion of channel attention and spatial attention; the first attention module is based on the spatial feature and the time characteristics After obtaining the preliminary fusion features, the second attention module uses the spatial features The time characteristics and the preliminary fusion features to obtain the intermediate features The spatiotemporal fusion remote sensing image generation module converts the intermediate features Input into the image reconstruction network to obtain the spatiotemporal fusion remote sensing image at time t2

Images