Deep fourier transform network-based remote sensing image spatio-temporal fusion method and system

Abstract

The application discloses a kind of based on deep Fourier Transform network's remote sensing image space-time fusion method and system, the specific method of the application is: the coarse image and fine image of reference date and the coarse image of prediction date are input into the generator trained, and the fine image of prediction date is output;Wherein, generator includes spectral coordination module and spatial information branch, spectral coordination module includes the amplitude-based Transform module and the phase-based Transform module.The method of the application is different from the traditional remote sensing image single spatial domain analysis, combines the advantages of frequency analysis in processing global feature, processing heterogeneous data and enhancing model generalization ability, and utilizes the self-attention mechanism of Transform model, effectively improves the authenticity of remote sensing image space-time fusion generation image, provides a new train of thought for remote sensing image space-time fusion task.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning and computer vision, and particularly relates to a method and system for spatiotemporal fusion of remote sensing images based on a deep Fourier Transformer network. Background Technology

[0002] With the continuous development of Earth observation technology, the demand for high-resolution remote sensing images with short time series is constantly increasing. High temporal and high spatial resolution remote sensing images play an important role in crop and forest monitoring, research on surface disaster dynamics, and detection of land cover change.

[0003] However, due to unavoidable technical and budgetary limitations, a single satellite often cannot simultaneously acquire high temporal and high spatial resolution remote sensing satellite imagery. Satellites providing high spatial resolution imagery typically have a long revisit period, while satellites with short revisit periods usually only provide low spatial resolution imagery. For example, the Moderate Resolution Imaging Spectroradiometer (MODIS) acquires remote sensing imagery with a spatial resolution of 250-1000 meters, but requires only one day. The Landsat-8 series of sensors acquires remote sensing imagery with a spatial resolution of 30 meters, but its time period is 16 days.

[0004] Spatiotemporal fusion combines Landsat and MODIS images to obtain high spatiotemporal resolution remote sensing images. While acknowledging the continuous changes of ground objects over time, it preserves the detailed features of these objects.

[0005] In recent years, a large number of spatiotemporal fusion algorithms have been developed based on the spatial domain. Spatial domain research is intuitive, allowing direct observation and interpretation of spatial features in images, such as terrain structure, land cover distribution, and environmental changes. Furthermore, spatial domain analysis excels in identifying and processing local features, particularly when focusing on detailed changes in specific areas such as urban planning or environmental monitoring. It can also effectively handle spatial relationships in images, including adjacency, overlap, and distance relationships, which is crucial for a deeper understanding of geographical phenomena and the spatial distribution of land cover. In handling certain real-time monitoring and response tasks, such as rapid monitoring of natural disasters, spatial domain analysis can provide rapid processing and decision-making capabilities, which is especially critical in emergency situations.

[0006] While spatial domain analysis has advantages in terms of intuitiveness and local feature processing, frequency domain analysis offers more advantages in handling nonlocal characteristics, improving computational efficiency, processing heterogeneous data, and enhancing model generalization capabilities, thus providing greater advantages in addressing certain specific problems in the spatiotemporal fusion of remote sensing images.

[0007] The frequency domain can reveal the periodicity and frequency-related information hidden in images, which is often difficult to detect in spatial domain analysis. Frequency domain analysis is particularly suitable for processing the non-local characteristics of images, such as overall trends and patterns. It can also adjust different frequency components to reconcile temporal and spatial differences, demonstrating its effectiveness in handling the heterogeneity between remote sensing data acquired at different resolutions and time points. Due to its wider applicability, frequency domain features help improve the model's generalization ability when processing different types of remote sensing data. Furthermore, frequency domain analysis helps suppress noise and enhance image features, playing a role in improving the overall quality and resolution of remote sensing images. Therefore, frequency domain analysis is a valuable direction in the spatiotemporal fusion of remote sensing images, providing a more comprehensive and in-depth perspective and strong support for extracting deeper information from images. Therefore, this invention mainly studies the spatiotemporal fusion of remote sensing images in the frequency domain, adjusting and fusing the amplitude and phase obtained through Fourier transform, and finally stitching together the spatial domain feature information. Summary of the Invention

[0008] The purpose of this invention is to solve the problems existing in the prior art and to provide a method and system for spatiotemporal fusion of remote sensing images based on deep Fourier Transformer networks.

[0009] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0010] In a first aspect, the present invention provides a method for spatiotemporal fusion of remote sensing images based on a deep Fourier Transformer network, comprising the following steps:

[0011] The coarse and fine images of the reference date, as well as the coarse image of the predicted date, are input into a trained generator, which outputs a fine image of the predicted date. During generator training, the generator and a discriminator form a remote sensing image spatiotemporal fusion model, jointly optimizing the model parameters. The generator is used to acquire the fine image of the predicted date, and the discriminator is used to distinguish between the fine image generated by the generator and the real fine image. The generator includes a spectral coordination module and a spatial information branch. The spectral coordination module includes an amplitude-based Transformer module and a phase-based Transformer module. The processing flow in the phase-based and amplitude-based Transformer modules is the same. Both the phase-based and amplitude-based Transformer modules include a multidimensional convolutional head transpose attention mechanism and a gated multidimensional convolutional feedforward network.

[0012] In the generator, the coarse image of the reference date undergoes multi-stage convolution through the first convolutional branch to extract features, the coarse image of the predicted date undergoes multi-stage convolution through the second convolutional branch to extract features, and the fine image of the reference date undergoes multi-stage convolution through the third convolutional branch to extract features. Each convolutional branch consists of multiple convolutional blocks with non-shared parameters cascaded sequentially. The first, second, and third convolutional branches contain the same number of convolutional blocks. Each stage of the convolutional block in each convolutional branch outputs an intermediate feature map. In the three convolutional branches, the intermediate feature maps output by the convolutional blocks of the same stage are used as the input of the spectrum coordination module of the corresponding stage, and the fine image of the reference date is passed to the spatial information branch. Then, the feature maps output by the spectrum coordination modules of each stage and the spatial information feature maps output by the spatial information branch are concatenated to obtain a fused feature map. The fused feature map is then passed through a standard convolutional layer to obtain the fine image of the predicted date.

[0013] Based on the above scheme, each step can be implemented in the following preferred manner.

[0014] Preferably, in each convolutional branch of the generator, each stage of the convolutional block contains a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function.

[0015] Preferably, the processing flow of the spectrum coordination module in each stage is the same. In the spectrum coordination module of the i-th stage, the first intermediate feature map output by the i-th stage convolutional block of the first convolutional branch, the second intermediate feature map output by the i-th stage convolutional block of the second convolutional branch, and the third intermediate feature map output by the i-th stage convolutional block of the third convolutional branch are obtained. The first intermediate feature map is subjected to Discrete Fourier Transform to obtain the first amplitude feature map and the first phase feature map. The second intermediate feature map is subjected to Discrete Fourier Transform to obtain the second amplitude feature map and the second phase feature map. The third intermediate feature map is subjected to Discrete Fourier Transform to obtain the third amplitude feature map and the third phase feature map. The first amplitude feature map and the third amplitude feature map are input into the amplitude-based Transformer module to reconstruct the corrected amplitude and obtain a new amplitude feature map. The first phase feature map and the second phase feature map are input into the phase-based Transformer module to reconstruct the corrected phase and obtain a new phase feature map. The new phase feature map and the new amplitude feature map are subjected to Inverse Discrete Fourier Transform to obtain the feature map output by the spectrum coordination module of the i-th stage.

[0016] Preferably, in the phase-based Transformer module or the amplitude-based Transformer module, the first input data and the second input data are processed by the multidimensional convolution head transpose attention mechanism to obtain a self-attention feature map, and the self-attention feature map is processed by the gated multidimensional convolution feedforward network to obtain a phase feature map or an amplitude feature map.

[0017] In the phase-based Transformer module, the first input data is the intermediate feature map output by each stage of the convolutional block in the first convolutional branch, and the second input data is the intermediate feature map output by each stage of the convolutional block in the second convolutional branch; in the amplitude-based Transformer module, the first input data is the intermediate feature map output by each stage of the convolutional block in the first convolutional branch, and the second input data is the intermediate feature map output by each stage of the convolutional block in the third convolutional branch.

[0018] Preferably, in the multidimensional convolutional head transposed attention mechanism, the first input data is normalized by a layer, and the normalized first input data is passed sequentially through a first convolutional layer and a first depth-wise convolutional layer to obtain value features. The normalized first input data is then passed sequentially through a second convolutional layer and a second depth-wise convolutional layer to obtain key features. The second input data is normalized by a layer, and the normalized second input data is passed sequentially through a third convolutional layer and a third depth-wise convolutional layer to obtain query features. Attention is calculated on the query features, key features, and value features to obtain a transposed attention map. The transposed attention map is passed through a fourth convolutional layer to obtain a new transposed attention map. The new transposed attention map is then residually connected to the second input data to obtain a self-attention feature map.

[0019] Preferably, in the gated multidimensional convolutional feedforward network, the self-attention feature map is normalized by layers. The normalized self-attention feature map is then passed sequentially through the fifth convolutional layer, the fourth depth-wise convolutional layer, and the GELU activation function to generate a gated map. The normalized self-attention feature map is then passed sequentially through the sixth convolutional layer and the fifth depth-wise convolutional layer to obtain a gated feature map. The gated map and the gated feature map are multiplied element-wise to obtain a weighted gated feature map. The weighted gated feature map is then passed through the seventh convolutional layer to obtain a depth-gated feature map. The depth-gated feature map is then residually connected to the self-attention feature map to obtain an output feature map, which is used as a phase feature map or an amplitude feature map.

[0020] Preferably, the kernel size of the first, second, third, fourth, fifth, sixth, and seventh convolutional layers is 1×1, and the kernel size of the first, second, third, fourth, and fifth depth-wise convolutional layers is 3×3.

[0021] As a preferred embodiment, the spatial information branch is composed of an eighth convolutional layer, an LRelu convolutional layer, and a ninth convolutional layer, with the kernel size of the eighth and ninth convolutional layers both being 3×3.

[0022] Preferably, the specific processing flow of the discriminator is as follows: the fine image generated by the generator and the real fine image are first passed through a tenth convolutional layer and a LeakyReLU activation function to obtain a first feature map. The first feature map is then processed through four first convolutional blocks in sequence to obtain a second feature map. Finally, the second feature map is passed through a twelfth convolutional layer and a sigmoid activation function to obtain the output result of the discriminator. Each first convolutional block contains an eleventh convolutional layer, batch normalization, and a LeakyReLU activation function. The kernel size of the tenth convolutional layer is 4×4, the kernel size of the eleventh convolutional layer is 4×4, and the kernel size of the twelfth convolutional layer is 4×4.

[0023] Secondly, the present invention provides a remote sensing image spatiotemporal fusion system based on a deep Fourier Transformer network, comprising:

[0024] The data acquisition module is used to acquire a coarse image of the reference date, a fine image of the reference date, and a coarse image of the predicted date;

[0025] The result acquisition module is used to input the coarse image of the reference date, the fine image of the reference date, and the coarse image of the predicted date into the trained generator, and output the fine image of the predicted date. During the training of the generator, the generator and a discriminator are combined to form a remote sensing image spatiotemporal fusion model, and the parameters of the remote sensing image spatiotemporal fusion model are jointly optimized. The generator is used to acquire the fine image of the predicted date, and the discriminator is used to distinguish between the fine image generated by the generator and the real fine image. The generator includes a spectrum coordination module and a spatial information branch. The spectrum coordination module includes an amplitude-based Transformer module and a phase-based Transformer module. The processing flow in the phase-based Transformer module and the amplitude-based Transformer module is the same. Both the phase-based Transformer module and the amplitude-based Transformer module include a multidimensional convolution head transposed attention mechanism and a gated multidimensional convolutional feedforward network.

[0026] In the generator, the coarse image of the reference date undergoes multi-stage convolution through the first convolutional branch to extract features, the coarse image of the predicted date undergoes multi-stage convolution through the second convolutional branch to extract features, and the fine image of the reference date undergoes multi-stage convolution through the third convolutional branch to extract features. Each convolutional branch consists of multiple convolutional blocks with non-shared parameters cascaded sequentially. The first, second, and third convolutional branches contain the same number of convolutional blocks. Each stage of the convolutional block in each convolutional branch outputs an intermediate feature map. In the three convolutional branches, the intermediate feature maps output by the convolutional blocks of the same stage are used as the input of the spectrum coordination module of the corresponding stage, and the fine image of the reference date is passed to the spatial information branch. Then, the feature maps output by the spectrum coordination modules of each stage and the spatial information feature maps output by the spatial information branch are concatenated to obtain a fused feature map. The fused feature map is then passed through a standard convolutional layer to obtain the fine image of the predicted date.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] This invention provides a method and system for spatiotemporal fusion of remote sensing images based on a deep Fourier Transform network. It introduces a spatiotemporal fusion model for remote sensing images, considering the significant differences in amplitude and phase between high-temporal-low-spatial and low-temporal-high-spatial images obtained from different sensors. By innovatively combining frequency domain information and the self-attention mechanism of the Transformer, it improves the ability to capture differences in spatial and temporal resolution between different images. Simultaneously, it fuses with local feature information extracted from the spatial domain, thereby improving the accuracy of comparing the generated image with the real image. These processing methods achieve better results. Attached Figure Description

[0029] Figure 1 This is an overall structural diagram of the remote sensing image spatiotemporal fusion model of the present invention;

[0030] Figure 2 This is a structural diagram of the phase-based Transformer module / amplitude-based Transformer module of the present invention;

[0031] Figure 3 This is a flowchart illustrating the training process of the method of the present invention;

[0032] Figure 4 This is a schematic diagram of the prediction results in an embodiment of the present invention;

[0033] Figure 5 This is a system block diagram of the present invention. Detailed Implementation

[0034] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0035] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0036] Spatiotemporal fusion is a problem that overcomes technological limitations by acquiring remote sensing images with both high temporal and spatial resolution. Its input consists of coarse and fine images for a reference date and a coarse image for a predicted date, used to generate a fine image for the predicted date. The significant resolution difference between the coarse and fine images, and the substantial temporal differences between images from different times, are two key considerations for spatiotemporal fusion. While spatial domain feature extraction is generally used to mitigate the large resolution issue, and spatial analysis offers advantages in intuitiveness and local feature processing, frequency domain analysis provides further advantages in handling nonlocal characteristics, improving computational efficiency, processing heterogeneous data, and enhancing model generalization capabilities, making it particularly beneficial for addressing specific problems in spatiotemporal fusion of remote sensing images.

[0037] Images with high temporal-low spatial resolution and low temporal-high spatial resolution obtained from different sensors exhibit significant differences in amplitude and phase. In spatiotemporal fusion, frequency domain interpolation can improve the capture of these significant spatial resolution differences, and the local feature information provided by the spatial domain branching module further enhances the quality of the generated image. Furthermore, images with temporal differences can capture phase transformation information of ground features, thereby improving the accuracy of comparison between the generated and real images. Therefore, in a preferred embodiment of the present invention, a remote sensing image spatiotemporal fusion method based on a deep Fourier Transformer network is proposed. Specifically, the coarse image (C1) and fine image (F1) of the reference date (t1) and the coarse image (C2) of the predicted date (t2) are input into a trained generator, and the fine image of the predicted date is output. During generator training, the generator and a discriminator constitute a remote sensing image spatiotemporal fusion model DFTNet, and the parameters of the remote sensing image spatiotemporal fusion model DFTNet are jointly optimized. The generator is used to obtain the fine image of the predicted date, and the discriminator is used to distinguish between the fine image generated by the generator and the real fine image. The generator includes a spectrum coordination module and a spatial information branch. The spectrum coordination module includes an amplitude-based Transformer module and a phase-based Transformer module. The processing flow in the phase-based Transformer module and the amplitude-based Transformer module is the same. Both the phase-based Transformer module and the amplitude-based Transformer module include a multidimensional convolution head transpose attention mechanism and a gated multidimensional convolution feedforward network.

[0038] In the generator, the coarse image of the reference date undergoes multi-stage convolution through the first convolutional branch to extract features, the coarse image of the predicted date undergoes multi-stage convolution through the second convolutional branch to extract features, and the fine image of the reference date undergoes multi-stage convolution through the third convolutional branch to extract features. Each convolutional branch consists of multiple convolutional blocks with non-shared parameters cascaded sequentially. The first, second, and third convolutional branches contain the same number of convolutional blocks. Each stage of the convolutional block in each convolutional branch outputs an intermediate feature map. In the three convolutional branches, the intermediate feature maps output by the convolutional blocks of the same stage are used as the input of the spectrum coordination module of the corresponding stage, and the fine image of the reference date is passed to the spatial information branch. Then, the feature maps output by the spectrum coordination modules of each stage and the spatial information feature maps output by the spatial information branch are concatenated to obtain a fused feature map. The fused feature map is then passed through a standard convolutional layer to obtain the fine image of the predicted date.

[0039] To address the technical challenge of effectively processing and fusing image features at different spatiotemporal resolutions and time periods using deep learning and computer vision technologies, this invention constructs a remote sensing image spatiotemporal fusion model based on a Deep-Fourier Transformer Network (DFTNet). The model structure is as follows: Figure 1 As shown.

[0040] The overall structure of the remote sensing image spatiotemporal fusion model DFTNet of this invention will be described in detail below, as follows: Figure 1 As shown, it comprises a generator and a discriminator, and is a type of Generative Adversarial Network (GAN). GANs have achieved excellent results in image generation, style transfer, and super-resolution reconstruction. They have also been applied in spatiotemporal fusion and have demonstrated good performance.

[0041] In the remote sensing image spatiotemporal fusion model DFTNet of this invention, the generator has three convolutional branches, each consisting of multiple convolutional blocks cascaded sequentially to extract features, and all three branches contain the same number of convolutional blocks. By stacking convolutional blocks, features at different scales are extracted in a local-to-global manner.

[0042] In this embodiment, each convolutional block in each convolutional branch contains a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function for feature extraction. Within each convolutional block, the 3×3 convolutional layer filters and extracts features from the input feature map, extracting features in a local-to-global manner to fully extract spatiotemporal variation information and ground details. The batch normalization layer helps accelerate the training process, reducing gradient vanishing or exploding problems, while improving the generalization ability of the remote sensing image spatiotemporal fusion model DFTNet. The ReLU activation function sets the negative part to zero, enhancing the expressive power of the remote sensing image spatiotemporal fusion model DFTNet.

[0043] In addition, in this embodiment, the remote sensing image spatiotemporal fusion model DFTNet is selected to have two stages, that is, the convolution branch is composed of two convolution blocks cascaded in sequence, which enables the remote sensing image spatiotemporal fusion model DFTNet to learn more complex features and patterns, increases the expressive power of the remote sensing image spatiotemporal fusion model DFTNet, and helps to better fit the data.

[0044] After the remote sensing image spatiotemporal fusion model DFTNet obtains feature maps of different scales through different stages, this invention introduces a spectrum coordination module with a Transformer structure to process the amplitude and phase of these feature maps at different scales, thus obtaining feature maps at different stages again. Furthermore, this invention introduces a spatial information branch to extract spatial information from the F1 image of the input reference date. Then, the feature maps obtained by the spectrum coordination module and the spatial information branch at each stage are concatenated along the channel number dimension to avoid information loss. Compared to simply stacking convolutional layers, this better preserves the feature information of each stage, resulting in a richer information feature representation. Finally, the concatenated feature map is processed by a 1×1 standard convolutional layer to map its feature channel number to the channel number of the real fine image, thereby generating a predicted image.

[0045] It should be noted that in this invention, the remote sensing image spatiotemporal fusion model DFTNet needs to jointly update the generator and discriminator parameters during training, but during inference, only the trained generator needs to be used. Its inputs during training and inference are the coarse image C1 and fine image F1 of the reference date and the coarse image C2 of the predicted date, to generate a high-resolution fine image F2′ (predicted image) of the predicted date.

[0046] Specifically, when training the remote sensing image spatiotemporal fusion model DFTNet, the generator takes a pair of coarse images of the reference date, a fine image, and a coarse image of the predicted date as input. It performs multiple convolutions on each image input, and feeds the feature maps obtained from each stage of convolution into the spectrum coordination module. At the same time, it feeds the fine image F1 into the spatial information branch. Finally, it concatenates the output of each stage of the spectrum coordination module with the spatial information feature map obtained from the spatial information branch. The concatenated fusion feature map is then subjected to a 1×1 convolution to map the number of feature channels to the number of channels of the real fine image F2, generating the fine image F2′ of the predicted date. The discriminator takes the detailed image F2′ of the predicted date and the real detailed image F2 as inputs and makes a judgment through a binary convolutional neural network. If the discriminator outputs True, it means that the detailed image F2′ of the predicted date and the real detailed image F2 are close. Conversely, if the discriminator outputs False, it means that the detailed image F2′ of the predicted date and the real detailed image F2 are far apart and not close. This allows the predicted image obtained by the remote sensing image spatiotemporal fusion model DFTNet to be infinitely close to the real image, thus constituting one training process.

[0047] It should also be noted that in this invention, the coarse image and the fine image are each acquired by a separate sensor, and the sampling points corresponding to the two images are different, thus resulting in different resolutions. For some dates, both coarse and fine images can be acquired simultaneously, while for others only a coarse image is acquired, without a fine image. In the description of this invention, dates on which both coarse and fine images can be acquired simultaneously are referred to as reference dates, and dates on which only a coarse image can be acquired without a fine image are referred to as prediction dates.

[0048] It should be noted that in this invention, the processing flow of the spectrum coordination module is the same at each stage. In the spectrum coordination module of the i-th stage, as follows... Figure 1 As shown, the first intermediate feature map of the output of the i-th stage convolutional block of the first convolutional branch is obtained. The second intermediate feature map output by the i-th stage convolutional block of the second convolutional branch. And the third intermediate feature map output by the i-th stage convolutional block of the third convolutional branch. For the first intermediate feature map Perform a discrete Fourier transform to obtain the first amplitude feature map and the first phase feature map, and then perform a second intermediate feature map. Perform a discrete Fourier transform to obtain the second amplitude feature map and the second phase feature map, and then perform a third intermediate feature map. Perform a Discrete Fourier Transform to obtain the third amplitude feature map and the third phase feature map. Input the first amplitude feature map and the third amplitude feature map into the amplitude-based Transformer module to reconstruct the corrected amplitude and obtain a new amplitude feature map. Input the first phase feature map and the second phase feature map into the phase-based Transformer module to reconstruct the corrected phase and obtain a new phase feature map. Perform an Inverse Discrete Fourier Transform on the new phase feature map and the new amplitude feature map to obtain the feature map output by the i-th stage spectrum coordination module.

[0049] The specific structure and principle of the aforementioned spectrum coordination module are described in detail below. In the spectrum coordination module, this invention introduces a Transformer model suitable for image reconstruction tasks, designed to process data composed of amplitude or phase obtained through Fourier transform. Based on this, an amplitude-based Transformer module and a phase-based Transformer module are constructed. In the amplitude-based Transformer module, it receives amplitude information from the coarse image (C1) and fine image (F1) corresponding to the reference date, i.e., the aforementioned first amplitude feature map and third amplitude feature map. In the phase-based Transformer module, it receives phase information from the coarse image (C1) corresponding to the reference date and the coarse image (C2) corresponding to the prediction date, i.e., the aforementioned first phase feature map and second phase feature map. By processing the amplitude and phase information through the multi-head self-attention mechanism and feedforward network in the original Transformer model, feature interaction and integration at different levels and scales can be achieved. Among them, the multi-head self-attention mechanism helps to capture global dependencies in the input information, enabling the remote sensing image spatiotemporal fusion model DFTNet to simultaneously focus on the feature representations of different parts, while the feedforward network can perform nonlinear transformation and extraction of features at each location, enhancing the representation capability of the remote sensing image spatiotemporal fusion model DFTNet.

[0050] It should be noted that in the phase-based or amplitude-based Transformer module of this invention, the first input data X and the second input data Y are processed through a multi-dimensional convolutional head transposed attention mechanism. This process involves projection onto the key (K), value (V), and query (Q) to simultaneously consider the relationships between different elements when calculating the attention weights. These relationships are modeled through the multi-dimensional convolutional head transposed attention mechanism, with each head capturing different aspects of the input sequence, ultimately resulting in a self-attention feature map. The self-attention feature map is passed through a gated multidimensional convolutional feedforward network to obtain the phase feature map or amplitude feature map Z.

[0051] It should be noted that in this invention, for the phase-based Transformer module, the information of the coarse image C1 of the reference date is needed to guide the generation of the coarse image C2 of the predicted date. Therefore, the first input data is the intermediate feature map output by each stage of the convolutional block of the first convolutional branch, and the second input data is the intermediate feature map output by each stage of the convolutional block of the second convolutional branch. That is, the first input data X corresponds to the coarse image C1 of the reference date, and the second input data Y corresponds to the coarse image C2 of the predicted date. In the amplitude-based Transformer module, the information of the coarse image C1 of the reference date is needed to guide the generation of the fine image F1 of the reference date. Therefore, the first input data is the intermediate feature map output by each stage of the convolutional block of the first convolutional branch, and the second input data is the intermediate feature map output by each stage of the convolutional block of the third convolutional branch. That is, the first input data X corresponds to the coarse image C1 of the reference date, and the second input data Y corresponds to the fine image F1 of the reference date.

[0052] In this embodiment, the intermediate processes of the amplitude-based Transformer module / phase-based Transformer module and the dimensions of each parameter are further described to enable those skilled in the art to better understand the principle of using the amplitude-based Transformer module / phase-based Transformer module.

[0053] Specifically, the structures of the amplitude-based Transformer module and the phase-based Transformer module are as follows: Figure 2 As shown, it includes a Multi-Dconv Head Transposed Attention (MDTA) mechanism and a Gated-Dconv Feed-Forward Network (GDFN). The original multi-head self-attention mechanism in the Transformer model has a high computational cost and is not suitable for application to high-resolution images. Therefore, this invention improves the multi-head self-attention mechanism to a multi-Dconv Head Transposed Attention mechanism, which is characterized by computing self-attention on each channel to implicitly encode global contextual information.

[0054] The multidimensional convolutional head transposed attention mechanism enriches the information extraction process, helping to more effectively capture long-range dependencies in high-resolution images and improving the ability of the DFTNet spatiotemporal fusion model to perceive global information. Simultaneously, this mechanism effectively reduces computational complexity, resulting in better scalability when processing large-scale data. Furthermore, the multidimensional convolutional head transposed attention mechanism uses depth-wise convolution operations to generate value features, key features, and query features when calculating the self-attention map, which helps emphasize local information.

[0055] In the multidimensional convolutional head transpose attention mechanism, the first input data X is first normalized by a layer LN(·). The normalized first input data LN(X) is then passed through the first convolutional layer and the first depth-wise convolutional layer to obtain the value features. In the formula, This represents the weight matrix of the first depth-wise convolutional layer. This represents the weight matrix of the first convolutional layer. The normalized first input data is then passed sequentially through the second convolutional layer and the second depth-wise convolutional layer to obtain the key features. In the formula, This represents the weight matrix of the second depth-wise convolutional layer. This represents the weight matrix of the second convolutional layer. The second input data Y is normalized through this layer, and the normalized second input data LN(Y) is then passed sequentially through the third convolutional layer and the third depth-wise convolutional layer to obtain the query features. In the formula, This represents the weight matrix of the third depth-wise convolutional layer. This represents the weight matrix of the third convolutional layer.

[0056] query features Key features and value characteristics The transposed attention map is obtained by calculating the attention level using the following formula:

[0057]

[0058] In the formula, α is a learnable scaling parameter applied to the Softmax function, which is used to obtain the transposed attention map.

[0059] The transposed attention map is passed through the fourth convolutional layer to obtain a new transposed attention map. This new transposed attention map is then residually concatenated with the second input data Y to obtain a self-attention feature map.

[0060]

[0061] In the formula, W p This represents the weight matrix of the fourth convolutional layer.

[0062] In this embodiment, the kernel size of the first, second, third, and fourth convolutional layers is 1×1, and the kernel size of the first, second, and third depth-wise convolutional layers is 3×3. Depth-wise convolutional layers perform convolution operations on each input channel separately with the kernel, unlike traditional convolution which combines all input channels together, thus reducing the number of parameters and computational cost.

[0063] Gated Multidimensional Convolutional Feedforward Networks (GDFNs) introduce gating mechanisms and depth-wise convolutions. Compared to the classic Transformer model, which uses a feedforward network for pixel-by-pixel feature operations, GDFNs use gating units to obtain the activation state of each pixel, use 1×1 convolutional layers to expand the number of channels, and then use 3×3 depth-wise convolutional layers and the GELU activation function to generate a gated map. GDFNs can be viewed as a filter for information, controlling the flow of information during the forward propagation of the network.

[0064] Gated multidimensional convolutional feedforward networks (GDFNs) allow each layer to focus on subtle details that complement other layers by controlling the information flow, which helps to filter and aggregate important feature information. This information filtering process can help the remote sensing image spatiotemporal fusion model DFTNet learn and utilize global contextual information more effectively, thereby enhancing DFTNet's ability to represent and generalize input data.

[0065] In gated multidimensional convolutional feedforward networks, self-attention feature maps are... After layer normalization, the normalized self-attention feature map The gated map is generated by sequentially passing through the fifth convolutional layer, the fourth depth-wise convolutional layer, and the GELU activation function. In the formula, This represents the weight matrix of the fifth convolutional layer. This represents the weight matrix of the fourth depth-wise convolutional layer. This represents the GELU nonlinear function. The normalized self-attention feature map is then passed sequentially through the sixth convolutional layer and the fifth depth-wise convolutional layer to obtain the gated feature map. In the formula, This represents the weight matrix of the sixth convolutional layer. This represents the weight matrix of the fifth depth-wise convolutional layer. The gated map and the gated feature map are multiplied element-wise to obtain the weighted gated feature map.

[0066]

[0067] The weighted gated feature map is passed through the seventh convolutional layer to obtain the deep gated feature map. In the formula, This represents the weight matrix of the seventh convolutional layer. It combines the depth-gated feature map with the self-attention feature map. Perform residual connection to obtain the output feature map Z, which can then be used as either a phase feature map or an amplitude feature map.

[0068]

[0069] In this embodiment, the kernel size of the fifth, sixth, and seventh convolutional layers is 1×1, and the kernel size of the fourth and fifth depth-wise convolutional layers is 3×3.

[0070] It should be noted that in this invention, the spatial information branch is formed by cascading the eighth convolutional layer, LRelu, and the ninth convolutional layer in sequence, with the kernel size of the eighth and ninth convolutional layers both being 3×3.

[0071] In this embodiment, the spatial information branch sequentially includes a 3×3 convolutional layer, an LReLU layer, and another 3×3 convolutional layer. Its input is a fine-grained image F1 of the reference date. After processing by the spatial information branch, the spatial information feature map output by the spatial information branch is obtained.

[0072] It should be noted that the specific processing flow in the discriminator of this invention is as follows: The fine image F2′ generated by the generator and the real fine image F2 are first passed through a tenth convolutional layer and a LeakyReLU activation function to obtain a first feature map. The first feature map is then processed through four first convolutional blocks in sequence to obtain a second feature map. Finally, the second feature map is passed through a twelfth convolutional layer and a sigmoid activation function to obtain the output result of the discriminator. Each first convolutional block contains an eleventh convolutional layer, batch normalization, and a LeakyReLU activation function. The kernel size of the tenth convolutional layer is 4×4, the kernel size of the eleventh convolutional layer is 4×4, and the kernel size of the twelfth convolutional layer is 4×4.

[0073] In this embodiment, the output of the discriminator is a value of 0 or 1, where 0 indicates that the image is a predicted image and 1 indicates that the image is a real image.

[0074] In summary, regarding significant resolution differences, images at different resolutions exhibit varying spectral characteristics due to sensor variations. This invention designs an amplitude-based Transformer module to reveal these spectral differences and reconstruct corrected amplitudes. Regarding temporal differences, subtle surface changes can be clearly revealed through phase information. This invention designs a phase-based Transformer module to more accurately reveal changes in ground structure and improve the resolution accuracy of time-series data. Furthermore, due to the global analytical characteristics of the frequency domain, this invention also incorporates a spatial information branch to compensate for the processing of local texture information, enhancing the ability of the remote sensing image spatiotemporal fusion model to perceive local details. Therefore, this invention provides an effective method for integrating Fourier transform data, making it applicable to various application scenarios requiring an understanding of frequency domain characteristics. Through in-depth experimental verification, the proposed remote sensing image spatiotemporal fusion model, DFTNet, demonstrates superior performance in processing Fourier transform data, particularly in capturing the complex relationship between amplitude and phase information.

[0075] The present invention will now demonstrate the application effect of the remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network described in the above embodiments on a specific dataset through a specific example, so as to facilitate understanding of the essence of the present invention.

[0076] Example

[0077] The specific implementation process of the remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network used in this embodiment is as described above and will not be repeated here.

[0078] The overall process in this embodiment can be divided into three stages: data preprocessing, model training, and image generation. Figure 3 As shown.

[0079] 1. Data Preprocessing

[0080] The evaluation was conducted using the CIA (Coleambally Irrigation Area) and LGC (lower Gwydir catchments (LGC)) open-source datasets. These two areas represent phenological and land cover changes, respectively. The CIA dataset contains 17 pairs of cloudless MODIS-Landsat images captured during 2001 and 2002; the LGC dataset contains 14 pairs of cloudless images captured during 2004 and 2005. Image sizes range from 1280×1792 to 3072×2056, and only the first four channels (B, G, R, and NIR) of all images were considered.

[0081] 2. Model Training

[0082] Step 1: Construct the training dataset and divide it into batches with a fixed batch size. Group two time-adjacent pairs of data into a single dataset. Use the last four pairs of data in each dataset as the validation set, and the rest as the training set.

[0083] Step 2: Train the remote sensing image spatiotemporal fusion model (DFTNet) using training samples from each batch. The specific structure of DFTNet is as described above and will not be repeated here. The training process is as follows: Figure 3 As shown, during training, both a generator and a discriminator are used simultaneously. The discriminator adjusts the network parameters of the remote sensing image spatiotemporal fusion model until all batches of the training dataset have participated in the training of the remote sensing image spatiotemporal fusion model. After reaching the specified number of iterations, the remote sensing image spatiotemporal fusion model converges, and training is complete.

[0084] 3. Image generation

[0085] The images from the test set are directly used as input to the trained remote sensing image spatiotemporal fusion model DFTNet generator, which ultimately generates a high-resolution image for the predicted date, thus achieving spatiotemporal fusion.

[0086] In this embodiment, the test results Figure 4 As shown, the remote sensing image spatiotemporal fusion model DFTNet of this invention can generate predicted images very well. Visually, EDCSTFN and GAN-STFM show their limitations in reconstructing images, particularly in terms of edge and texture details, resulting in blurred transformations of the real scene in the generated images. They still have room for improvement in capturing high-frequency details of images, especially in edge sharpness and texture clarity. The remote sensing image spatiotemporal fusion model DFTNet has made significant progress in this regard, exhibiting higher accuracy and clarity in detail generation. This advantage makes the remote sensing image spatiotemporal fusion model DFTNet particularly effective in image reconstruction tasks with rich detail or high contrast.

[0087] In processing spectral information, the DFTNet spatiotemporal fusion model for remote sensing images has successfully captured rich spectral information through in-depth research in the frequency domain. Compared with MS-Fusion's performance in color reproduction, DFTNet achieves more precise acquisition and utilization of spectral information through frequency domain analysis, thus achieving better results in color fidelity and spectral accuracy. This makes the DFTNet spatiotemporal fusion model for remote sensing images particularly important in fields requiring high spectral fidelity, such as remote sensing image analysis.

[0088] In terms of spatial detail reproduction, the DFTNet spatiotemporal fusion model for remote sensing images also demonstrates superior performance. It exhibits a more significant advantage over SwinSTFM and MLFF-GAN in restoring the spatial accuracy of images. This capability of the DFTNet spatiotemporal fusion model is particularly valuable for applications requiring precise detail resolution, such as fine terrain mapping, urban planning, and precision agricultural monitoring.

[0089] In comparison with G-TFFN, the spatiotemporal fusion model DFTNet for remote sensing images shows no significant difference in image quality. However, DFTNet demonstrates some improvement in noise handling. This improvement likely stems from DFTNet's greater susceptibility to noise in the frequency domain, allowing it to more effectively separate noise components when extracting image features.

[0090] It should also be noted that the remote sensing image spatiotemporal fusion method based on Deep Fourier Transformer networks in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a remote sensing image spatiotemporal fusion system based on Deep Fourier Transformer networks, corresponding to the remote sensing image spatiotemporal fusion method based on Deep Fourier Transformer networks provided in the above embodiments, such as... Figure 5 As shown, it includes:

[0091] The data acquisition module is used to acquire a coarse image of the reference date, a fine image of the reference date, and a coarse image of the predicted date;

[0092] The result acquisition module is used to input the coarse image of the reference date, the fine image of the reference date, and the coarse image of the predicted date into the trained generator, and output the fine image of the predicted date. During the training of the generator, the generator and a discriminator are combined to form a remote sensing image spatiotemporal fusion model, and the parameters of the remote sensing image spatiotemporal fusion model are jointly optimized. The generator is used to acquire the fine image of the predicted date, and the discriminator is used to distinguish between the fine image generated by the generator and the real fine image. The generator includes a spectrum coordination module and a spatial information branch. The spectrum coordination module includes an amplitude-based Transformer module and a phase-based Transformer module. The processing flow in the phase-based Transformer module and the amplitude-based Transformer module is the same. Both the phase-based Transformer module and the amplitude-based Transformer module include a multidimensional convolution head transposed attention mechanism and a gated multidimensional convolutional feedforward network.

[0093] In the generator, the coarse image of the reference date undergoes multi-stage convolution through the first convolutional branch to extract features, the coarse image of the predicted date undergoes multi-stage convolution through the second convolutional branch to extract features, and the fine image of the reference date undergoes multi-stage convolution through the third convolutional branch to extract features. Each convolutional branch consists of multiple convolutional blocks with non-shared parameters cascaded sequentially. The first, second, and third convolutional branches contain the same number of convolutional blocks. Each stage of the convolutional block in each convolutional branch outputs an intermediate feature map. In the three convolutional branches, the intermediate feature maps output by the convolutional blocks of the same stage are used as the input of the spectrum coordination module of the corresponding stage, and the fine image of the reference date is passed to the spatial information branch. Then, the feature maps output by the spectrum coordination modules of each stage and the spatial information feature maps output by the spatial information branch are concatenated to obtain a fused feature map. The fused feature map is then passed through a standard convolutional layer to obtain the fine image of the predicted date.

[0094] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A spatiotemporal fusion method for remote sensing images based on a deep Fourier Transformer network, characterized in that, Includes the following steps: The coarse and fine images of the reference date, as well as the coarse image of the predicted date, are input into a trained generator, which outputs a fine image of the predicted date. During generator training, the generator and a discriminator form a remote sensing image spatiotemporal fusion model, jointly optimizing the model parameters. The generator is used to acquire the fine image of the predicted date, and the discriminator is used to distinguish between the fine image generated by the generator and the real fine image. The generator includes a spectral coordination module and a spatial information branch. The spectral coordination module includes an amplitude-based Transformer module and a phase-based Transformer module. The processing flow in the phase-based and amplitude-based Transformer modules is the same. Both the phase-based and amplitude-based Transformer modules include a multidimensional convolutional head transpose attention mechanism and a gated multidimensional convolutional feedforward network. In the generator, the coarse image of the reference date undergoes multi-stage convolution through the first convolutional branch to extract features, the coarse image of the predicted date undergoes multi-stage convolution through the second convolutional branch to extract features, and the fine image of the reference date undergoes multi-stage convolution through the third convolutional branch to extract features. Each convolutional branch consists of multiple convolutional blocks with non-shared parameters cascaded sequentially. The first, second, and third convolutional branches contain the same number of convolutional blocks. Each stage of the convolutional block in each convolutional branch outputs an intermediate feature map. In the three convolutional branches, the intermediate feature maps output by the convolutional blocks of the same stage are used as the input of the spectrum coordination module of the corresponding stage, and the fine image of the reference date is passed to the spatial information branch. Then, the feature maps output by the spectrum coordination modules of each stage and the spatial information feature maps output by the spatial information branch are concatenated to obtain a fused feature map. The fused feature map is then passed through a standard convolutional layer to obtain the fine image of the predicted date.

2. The remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network as described in claim 1, characterized in that, In each convolutional branch of the generator, each stage of the convolutional block contains a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function.

3. The remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network as described in claim 1, characterized in that, The processing flow of the spectrum coordination module at each stage is the same. In the spectrum coordination module of the i-th stage, the first intermediate feature map output by the i-th stage convolutional block of the first convolutional branch, the second intermediate feature map output by the i-th stage convolutional block of the second convolutional branch, and the third intermediate feature map output by the i-th stage convolutional block of the third convolutional branch are obtained. The first intermediate feature map is subjected to Discrete Fourier Transform to obtain the first amplitude feature map and the first phase feature map. The second intermediate feature map is subjected to Discrete Fourier Transform to obtain the second amplitude feature map and the second phase feature map. The third intermediate feature map is subjected to Discrete Fourier Transform to obtain the third amplitude feature map and the third phase feature map. The first amplitude feature map and the third amplitude feature map are input into the amplitude-based Transformer module to reconstruct the corrected amplitude and obtain a new amplitude feature map. The first phase feature map and the second phase feature map are input into the phase-based Transformer module to reconstruct the corrected phase and obtain a new phase feature map. The new phase feature map and the new amplitude feature map are subjected to Inverse Discrete Fourier Transform to obtain the feature map output by the spectrum coordination module of the i-th stage.

4. The remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network as described in claim 1, characterized in that, In the phase-based or amplitude-based Transformer module, the first and second input data are processed by the multidimensional convolution head transpose attention mechanism to obtain the self-attention feature map. The self-attention feature map is then processed by the gated multidimensional convolution feedforward network to obtain the phase feature map or amplitude feature map. In the phase-based Transformer module, the first input data is the intermediate feature map output by each stage of the convolutional block in the first convolutional branch, and the second input data is the intermediate feature map output by each stage of the convolutional block in the second convolutional branch; in the amplitude-based Transformer module, the first input data is the intermediate feature map output by each stage of the convolutional block in the first convolutional branch, and the second input data is the intermediate feature map output by each stage of the convolutional block in the third convolutional branch.

5. The remote sensing image spatiotemporal fusion method based on a deep Fourier Transformer network as described in claim 4, characterized in that, In the multidimensional convolutional head transposed attention mechanism, the first input data is normalized through a layer, and the normalized first input data is then passed through the first convolutional layer and the first depth-wise convolutional layer to obtain value features. The normalized first input data is then passed through the second convolutional layer and the second depth-wise convolutional layer to obtain key features. The second input data is normalized through a layer, and the normalized second input data is then passed through the third convolutional layer and the third depth-wise convolutional layer to obtain query features. Attention is calculated on the query features, key features, and value features to obtain a transposed attention map. The transposed attention map is then passed through the fourth convolutional layer to obtain a new transposed attention map. The new transposed attention map is then residually connected to the second input data to obtain a self-attention feature map.

6. The remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network as described in claim 5, characterized in that, In the gated multidimensional convolutional feedforward network, the self-attention feature map is normalized by layers. The normalized self-attention feature map is then passed sequentially through the fifth convolutional layer, the fourth depth-wise convolutional layer, and the GELU activation function to generate a gated map. The normalized self-attention feature map is then passed sequentially through the sixth convolutional layer and the fifth depth-wise convolutional layer to obtain the gated feature map. The gated map and the gated feature map are multiplied element-wise to obtain a weighted gated feature map. The weighted gated feature map is then passed through the seventh convolutional layer to obtain a depth-gated feature map. The depth-gated feature map is then residually connected to the self-attention feature map to obtain the output feature map, which is used as either a phase feature map or an amplitude feature map.

7. The remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network as described in claim 6, characterized in that, The kernel size of the first, second, third, fourth, fifth, sixth, and seventh convolutional layers is 1×1, while the kernel size of the first, second, third, fourth, and fifth depth-wise convolutional layers is 3×3.

8. The remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network as described in claim 1, characterized in that, The spatial information branch is composed of the eighth convolutional layer, LRelu, and the ninth convolutional layer in sequence. The kernel size of the eighth and ninth convolutional layers is 3×3.

9. The remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network as described in claim 1, characterized in that, The specific processing flow of the discriminator is as follows: The fine image generated by the generator and the real fine image are first passed through a tenth convolutional layer and a LeakyReLU activation function to obtain a first feature map. The first feature map is then processed through four first convolutional blocks in sequence to obtain a second feature map. Finally, the second feature map is passed through a twelfth convolutional layer and a sigmoid activation function to obtain the output result of the discriminator. Each first convolutional block contains an eleventh convolutional layer, batch normalization, and a LeakyReLU activation function. The kernel size of the tenth convolutional layer is 4×4, the kernel size of the eleventh convolutional layer is 4×4, and the kernel size of the twelfth convolutional layer is 4×4.

10. A spatiotemporal fusion system for remote sensing images based on a deep Fourier Transformer network, characterized in that, include: The data acquisition module is used to acquire a coarse image of the reference date, a fine image of the reference date, and a coarse image of the predicted date; The result acquisition module is used to input the coarse image of the reference date, the fine image of the reference date, and the coarse image of the predicted date into the trained generator, and output the fine image of the predicted date. During the training of the generator, the generator and a discriminator are combined to form a remote sensing image spatiotemporal fusion model, and the parameters of the remote sensing image spatiotemporal fusion model are jointly optimized. The generator is used to acquire the fine image of the predicted date, and the discriminator is used to distinguish between the fine image generated by the generator and the real fine image. The generator includes a spectrum coordination module and a spatial information branch. The spectrum coordination module includes an amplitude-based Transformer module and a phase-based Transformer module. The processing flow in the phase-based Transformer module and the amplitude-based Transformer module is the same. Both the phase-based Transformer module and the amplitude-based Transformer module include a multidimensional convolution head transposed attention mechanism and a gated multidimensional convolutional feedforward network. In the generator, the coarse image of the reference date undergoes multi-stage convolution through the first convolutional branch to extract features, the coarse image of the predicted date undergoes multi-stage convolution through the second convolutional branch to extract features, and the fine image of the reference date undergoes multi-stage convolution through the third convolutional branch to extract features. Each convolutional branch consists of multiple convolutional blocks with non-shared parameters cascaded sequentially. The first, second, and third convolutional branches contain the same number of convolutional blocks. Each stage of the convolutional block in each convolutional branch outputs an intermediate feature map. In the three convolutional branches, the intermediate feature maps output by the convolutional blocks of the same stage are used as the input of the spectrum coordination module of the corresponding stage, and the fine image of the reference date is passed to the spatial information branch. Then, the feature maps output by the spectrum coordination modules of each stage and the spatial information feature maps output by the spatial information branch are concatenated to obtain a fused feature map. The fused feature map is then passed through a standard convolutional layer to obtain the fine image of the predicted date.

Images