Remote sensing image space-time fusion method and system based on deep Fourier Transform network

Through the deep Fourier Transformer network spatiotemporal fusion method, combining spectrum and airspace information, the resolution difference and noise problems in remote sensing image fusion are solved, image quality and contrast are improved, and are suitable for applications with high spectral and rich details.

CN118941903AActive Publication Date: 2024-11-12ZHEJIANG UNIV

Patent Information

Application Number
CN202410965097.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-12
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

The existing technology is difficult to effectively integrate remote sensing images with high time and high spatial resolution, resulting in the problems of information loss and insufficient noise suppression capabilities in crop monitoring, surface disaster research and land cover change detection.

Method used

The space-time fusion method based on the deep Fourier Transformer network is adopted, and the spectrum coordination module and the airspace information branch are combined, and the amplitude and phase processing and reconstruction are used to fuse image features with different time and spatial resolutions.

Benefits of technology

It improves the capture ability of the spatial and temporal resolution differences of the image, improves the accuracy of the contrast between the generated image and the real image, enhances noise suppression and image quality, and is suitable for application scenarios with high spectral fidelity and rich details.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941903A_ABST
    Figure CN118941903A_ABST
Patent Text Reader

Abstract

The invention discloses a remote sensing image space-time fusion method and system based on a deep Fourier Transform network, and the method specifically comprises the steps: inputting a coarse image and a fine image of a reference date and a coarse image of a prediction date into a trained generator, and outputting the fine image of the prediction date; wherein the generator comprises a frequency spectrum coordination module and an airspace information branch, and the frequency spectrum coordination module comprises an amplitude-based Transform module and a phase-based Transform module. The method is different from traditional remote sensing image single airspace analysis, the advantages of frequency analysis in global feature processing, heterogeneous data processing and model generalization ability enhancement are combined, the self-attention mechanism of the Transform model is utilized, the authenticity of the remote sensing image space-time fusion generated image is effectively improved, and the method is suitable for being applied to the remote sensing image space-time fusion generation. And a new thought is provided for a remote sensing image space-time fusion task.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and computer vision, and in particular to a remote sensing image spatiotemporal fusion method and system based on a deep Fourier Transformer network. Background Art

[0002] With the continuous development of earth observation technology, the demand for high-resolution remote sensing images of short time series is increasing. Remote sensing images with high temporal and spatial resolution play an important role in crop and forest monitoring, surface disaster dynamics research, and land cover change detection.

[0003] However, due to some unavoidable technical and budgetary limitations, a single satellite is often unable to simultaneously obtain remote sensing satellite images with high temporal and spatial resolutions. Satellites that provide high spatial resolution images usually have a longer revisit period, while satellites with short revisit periods can usually only provide low spatial resolution images. For example, the spatial resolution of remote sensing images obtained by the Moderate Resolution Imaging Spectroradiometer (MODIS) is 250-1000 meters, but it only takes 1 day. The spatial resolution of remote sensing images obtained by the Landsat-8 series sensors is 30 meters, but its temporal period is 16 days.

[0004] Spatiotemporal fusion fuses Landsat images and MODIS images to obtain remote sensing images with high spatiotemporal resolution. It preserves the detailed features of ground objects while meeting the continuous changes of ground objects over time.

[0005] In recent years, a large number of spatiotemporal fusion algorithms have been developed based on the airspace. Research on the airspace is intuitive and can directly observe and interpret the spatial characteristics of the image, such as terrain structure, distribution of objects, and environmental changes. In addition, airspace analysis excels in the recognition and processing of local features, especially when it is necessary to focus on detailed changes in specific areas such as urban planning or environmental monitoring. At the same time, it can also effectively process spatial relationships in images, including adjacency, overlap, and distance relationships, which is crucial for a deep understanding of geographical phenomena and the spatial distribution of surface cover. When dealing with certain real-time monitoring and response tasks, such as rapid monitoring of natural disasters, airspace analysis can provide rapid processing and decision-making capabilities, which is particularly critical in emergency situations.

[0006] Although spatial domain analysis has advantages in intuitiveness and local feature processing, considering the advantages of frequency domain analysis in processing non-local characteristics, improving computational efficiency, processing heterogeneous data, and enhancing model generalization capabilities, it provides more advantages when dealing with some specific problems of spatiotemporal fusion of remote sensing images.

[0007] The frequency domain can reveal the periodicity and frequency-related information hidden in the image, which is often not easy to detect in the spatial domain analysis. Frequency domain analysis is particularly suitable for processing the non-local characteristics of images, such as overall trends and patterns. At the same time, it can adjust different frequency components to coordinate the differences in time and space, and also show its effect when processing the heterogeneity between remote sensing data collected at different resolutions and different time points. Since the frequency domain features have a wider applicability, it helps to improve the generalization ability of the model in processing different types of remote sensing data. In addition, frequency domain analysis helps to suppress noise and enhance image features, and plays a certain role in improving the overall quality and resolution of remote sensing images. Therefore, frequency domain analysis is a valuable direction in the spatiotemporal fusion of remote sensing images, which can provide a more comprehensive and in-depth perspective, and provide strong support for extracting deeper information from images. Therefore, the present invention mainly studies the spatiotemporal fusion of remote sensing images in the frequency domain, adjusts and fuses the amplitude and phase obtained by Fourier transform, and finally splices and fuses the spatial feature information. Summary of the invention

[0008] The purpose of the present invention is to solve the problems existing in the prior art and provide a remote sensing image spatiotemporal fusion method and system based on a deep Fourier Transformer network.

[0009] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0010] In a first aspect, the present invention provides a remote sensing image spatiotemporal fusion method based on a deep Fourier Transformer network, comprising the following steps:

[0011] Input the coarse image and fine image of the reference date and the coarse image of the predicted date into the trained generator, and output the fine image of the predicted date; when training the generator, the generator and a discriminator constitute a remote sensing image spatiotemporal fusion model, and jointly optimize the parameters of the remote sensing image spatiotemporal fusion model, the generator is used to obtain the fine image of the predicted date, and the discriminator is used to distinguish the fine image generated by the generator from the real fine image, wherein the generator includes a spectrum coordination module and a spatial information branch, the spectrum coordination module includes an amplitude-based Transformer module and a phase-based Transformer module; the processing flow in the phase-based Transformer module and the amplitude-based Transformer module is the same, and the phase-based Transformer module or the amplitude-based Transformer module both include a multi-dimensional convolution head transposition attention mechanism and a gated multi-dimensional convolution feedforward network;

[0012] In the generator, the coarse image of the reference date is subjected to multi-stage convolution through the first convolution branch to extract features, the coarse image of the predicted date is subjected to multi-stage convolution through the second convolution branch to extract features, and the fine image of the reference date is subjected to multi-stage convolution through the third convolution branch to extract features. Each convolution branch is formed by sequentially cascading a plurality of convolution blocks whose parameters are not shared. The first convolution branch, the second convolution branch and the third convolution branch contain the same number of convolution blocks. The convolution blocks of each stage in each convolution branch output an intermediate feature map. In the three convolution branches, the intermediate feature maps output by the convolution blocks of the same stage are used as the input of the spectrum coordination module of the corresponding stage, and the fine image of the reference date is passed into the spatial information branch. Then, the feature maps output by the spectrum coordination modules of each stage and the spatial information feature maps output by the spatial information branch are spliced ​​to obtain a fused feature map. The fused feature map is passed through a standard convolution layer to obtain a fine image of the predicted date.

[0013] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0014] Preferably, in each convolution branch of the generator, the convolution block at each stage includes a 3×3 convolution layer, a batch normalization layer and a ReLU activation function.

[0015] Preferably, the processing flow of the spectrum coordination module in each stage is the same. In the spectrum coordination module of the i-th stage, a first intermediate feature map output by the i-th stage convolution block of the first convolution branch, a second intermediate feature map output by the i-th stage convolution block of the second convolution branch, and a third intermediate feature map output by the i-th stage convolution block of the third convolution branch are obtained, a discrete Fourier transform is performed on the first intermediate feature map to obtain a first amplitude feature map and a first phase feature map, a discrete Fourier transform is performed on the second intermediate feature map to obtain a second amplitude feature map and a second phase feature map, a discrete Fourier transform is performed on the third intermediate feature map to obtain a third amplitude feature map and a third phase feature map, the first amplitude feature map and the third amplitude feature map are input into an amplitude-based Transformer module, the corrected amplitude is reconstructed to obtain a new amplitude feature map, the first phase feature map and the second phase feature map are input into a phase-based Transformer module, the corrected phase is reconstructed to obtain a new phase feature map, and the new phase feature map and the new amplitude feature map are inversely discrete Fourier transformed to obtain a feature map output by the i-th stage spectrum coordination module.

[0016] Preferably, in a phase-based Transformer module or an amplitude-based Transformer module, the first input data and the second input data are calculated by a multidimensional convolution head transposition attention mechanism to obtain a self-attention feature map, and the self-attention feature map is passed through a gated multidimensional convolution feedforward network to obtain a phase feature map or an amplitude feature map;

[0017] In the phase-based Transformer module, the first input data is the intermediate feature map output by the convolution block of each stage of the first convolution branch, and the second input data is the intermediate feature map output by the convolution block of each stage of the second convolution branch; in the amplitude-based Transformer module, the first input data is the intermediate feature map output by the convolution block of each stage of the first convolution branch, and the second input data is the intermediate feature map output by the convolution block of each stage of the third convolution branch.

[0018] Preferably, in the transposed attention mechanism of the multi-dimensional convolutional head, the first input data is layer-normalized, the normalized first input data is sequentially passed through the first convolutional layer and the first depth-wise convolutional layer to obtain value features, the normalized first input data is sequentially passed through the second convolutional layer and the second depth-wise convolutional layer to obtain key features, the second input data is layer-normalized, the normalized second input data is sequentially passed through the third convolutional layer and the third depth-wise convolutional layer to obtain query features, attention is calculated on the query features, key features and value features to obtain a transposed attention map, the transposed attention map is passed through the fourth convolutional layer to obtain a new transposed attention map, the new transposed attention map is residually connected with the second input data to obtain a self-attention feature map.

[0019] Preferably, in a gated multidimensional convolutional feedforward network, the self-attention feature map is layer-normalized, the normalized self-attention feature map is sequentially passed through the fifth convolution layer, the fourth depth-wise convolution layer and the GELU activation function to generate a gated mapping map, the normalized self-attention feature map is sequentially passed through the sixth convolution layer and the fifth depth-wise convolution layer to obtain a gated feature map, the gated mapping map and the gated feature map are element-wise multiplied to obtain a weighted gated feature map, the weighted gated feature map is passed through the seventh convolution layer to obtain a depth gated feature map, the depth gated feature map is residually connected with the self-attention feature map to obtain an output feature map and use it as a phase feature map or an amplitude feature map.

[0020] Preferably, the convolution kernel sizes of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer, the sixth convolution layer and the seventh convolution layer are all 1×1, and the convolution kernel sizes of the first depth-wise convolution layer, the second depth-wise convolution layer, the third depth-wise convolution layer, the fourth depth-wise convolution layer and the fifth depth-wise convolution layer are all 3×3.

[0021] Preferably, the spatial information branch is formed by sequentially cascading the eighth convolutional layer, LRelu and the ninth convolutional layer, and the convolution kernel sizes of the eighth convolutional layer and the ninth convolutional layer are both 3×3.

[0022] Preferably, the specific processing flow of the discriminator is as follows: the fine image generated by the generator and the real fine image are first passed through a tenth convolution layer and a LeakyReLU activation function to obtain a first feature map, the first feature map is processed in turn through four first convolution blocks to obtain a second feature map, and finally the second feature map is passed through a twelfth convolution layer and a sigmoid activation function to obtain the output result of the discriminator, wherein each first convolution block includes an eleventh convolution layer, batch normalization and a LeakyReLU activation function, the convolution kernel size of the tenth convolution layer is 4×4, the convolution kernel size of the eleventh convolution layer is 4×4, and the convolution kernel size of the twelfth convolution layer is 4×4.

[0023] In a second aspect, the present invention provides a remote sensing image spatiotemporal fusion system based on a deep Fourier Transformer network, comprising:

[0024] A data acquisition module, used to acquire a coarse image of a reference date, a fine image of a reference date, and a coarse image of a predicted date;

[0025] The result acquisition module is used to input the coarse image of the reference date, the fine image of the reference date and the coarse image of the predicted date into the trained generator, and output the fine image of the predicted date; when training the generator, the generator and a discriminator are used to form a remote sensing image spatiotemporal fusion model, and the parameters of the remote sensing image spatiotemporal fusion model are jointly optimized, the generator is used to obtain the fine image of the predicted date, and the discriminator is used to distinguish the fine image generated by the generator from the real fine image, wherein the generator includes a spectrum coordination module and a spatial information branch, the spectrum coordination module includes an amplitude-based Transformer module and a phase-based Transformer module; the processing flow in the phase-based Transformer module and the amplitude-based Transformer module is the same, and the phase-based Transformer module or the amplitude-based Transformer module both include a multi-dimensional convolution head transposition attention mechanism and a gated multi-dimensional convolution feedforward network;

[0026] In the generator, the coarse image of the reference date is subjected to multi-stage convolution through the first convolution branch to extract features, the coarse image of the predicted date is subjected to multi-stage convolution through the second convolution branch to extract features, and the fine image of the reference date is subjected to multi-stage convolution through the third convolution branch to extract features. Each convolution branch is formed by sequentially cascading a plurality of convolution blocks whose parameters are not shared. The first convolution branch, the second convolution branch and the third convolution branch contain the same number of convolution blocks. The convolution blocks of each stage in each convolution branch output an intermediate feature map. In the three convolution branches, the intermediate feature maps output by the convolution blocks of the same stage are used as the input of the spectrum coordination module of the corresponding stage, and the fine image of the reference date is passed into the spatial information branch. Then, the feature maps output by the spectrum coordination modules of each stage and the spatial information feature maps output by the spatial information branch are spliced ​​to obtain a fused feature map. The fused feature map is passed through a standard convolution layer to obtain a fine image of the predicted date.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention provides a remote sensing image spatiotemporal fusion method and system based on a deep Fourier Transformer network, introduces a remote sensing image spatiotemporal fusion model, takes into account the huge differences in amplitude and phase between high-time and low-space and low-time and high-space images obtained by different sensors in the spatiotemporal fusion problem, and improves the ability to capture the differences in spatial and temporal resolution of different images by innovatively combining frequency domain information and the self-attention mechanism of the Transformer. At the same time, it is fused with the local feature information extracted from the spatial domain to improve the accuracy of the comparison between the generated image and the real image. These treatments have achieved better results. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is the overall structure diagram of the remote sensing image spatiotemporal fusion model of the present invention;

[0030] Figure 2 It is a structural diagram of a phase-based Transformer module / amplitude-based Transformer module of the present invention;

[0031] Figure 3 is a training flow chart of the method of the present invention;

[0032] Figure 4 A schematic diagram of prediction results according to an embodiment of the present invention;

[0033] Figure 5 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0035] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0036] The problem of spatiotemporal fusion is to break through the limitations of technology and obtain remote sensing images with high temporal and spatial resolution at the same time. Its input is the coarse and fine image of the reference date and the coarse image of the predicted date to generate the fine image of the predicted date. There is a huge resolution difference between the coarse image and the fine image, and there is a large time difference between images at different times. These are two key points that need to be considered in the spatiotemporal fusion problem. The spatiotemporal fusion task generally uses spatial domain feature extraction to alleviate the huge resolution problem. Although spatial domain analysis has advantages in intuitiveness and local feature processing, considering the advantages of frequency domain analysis in processing non-local characteristics, improving computational efficiency, processing heterogeneous data, and enhancing model generalization capabilities, it provides more advantages when dealing with some specific problems of spatiotemporal fusion of remote sensing images.

[0037] Images with high temporal and low spatial resolution and low temporal and high spatial resolution obtained by different sensors show obvious differences in amplitude and phase. In terms of the huge difference in spatial resolution in terms of spatiotemporal fusion, the difference processing in the frequency domain can improve the capture of image resolution differences, and the local feature information provided by the spatial domain branch module can improve the quality of the generated image. In addition, images with time differences can capture the transformation information of the structure of the object in phase, thereby improving the accuracy of the comparison between the generated image and the real image. Therefore, in a preferred implementation of the present invention, a remote sensing image spatiotemporal fusion method based on a deep Fourier Transformer network is proposed, specifically: a coarse image (C1) and a fine image (F1) of a reference date (t1) and a coarse image (C2) of a predicted date (t2) are input into a trained generator, and a fine image of the predicted date is output; when training the generator, the generator and a discriminator constitute a remote sensing image spatiotemporal fusion model DFTNet, and the parameters of the remote sensing image spatiotemporal fusion model DFTNet are jointly optimized, the generator is used to obtain a fine image of the predicted date, and the discriminator is used to distinguish the fine image generated by the generator from the real fine image; wherein the generator includes a spectrum coordination module and a spatial information branch, and the spectrum coordination module includes an amplitude-based Transformer module and a phase-based Transformer module. The processing flow in the phase-based Transformer module and the amplitude-based Transformer module is the same, and the phase-based Transformer module or the amplitude-based Transformer module both include a multi-dimensional convolutional head transposition attention mechanism and a gated multi-dimensional convolutional feedforward network.

[0038] In the generator, the coarse image of the reference date is subjected to multi-stage convolution through the first convolution branch to extract features, the coarse image of the predicted date is subjected to multi-stage convolution through the second convolution branch to extract features, and the fine image of the reference date is subjected to multi-stage convolution through the third convolution branch to extract features. Each convolution branch is formed by sequentially cascading a plurality of convolution blocks whose parameters are not shared. The first convolution branch, the second convolution branch and the third convolution branch contain the same number of convolution blocks. The convolution blocks of each stage in each convolution branch output an intermediate feature map. In the three convolution branches, the intermediate feature maps output by the convolution blocks of the same stage are used as the input of the spectrum coordination module of the corresponding stage, and the fine image of the reference date is passed into the spatial information branch. Then, the feature maps output by the spectrum coordination modules of each stage and the spatial information feature maps output by the spatial information branch are spliced ​​to obtain a fused feature map. The fused feature map is passed through a standard convolution layer to obtain a fine image of the predicted date.

[0039] In order to solve the technical problem of how to use deep learning and computer vision technology to process and effectively fuse image features at different spatiotemporal resolutions and at different times, a remote sensing image spatiotemporal fusion model based on a deep Fourier Transformer network (DFTNet) is constructed in the method of the present invention. The model structure is as follows Figure 1 shown.

[0040] The overall structure of the remote sensing image spatiotemporal fusion model DFTNet of the present invention is described in detail below. Figure 1 As shown in the figure, it contains a generator and a discriminator, which is a generative adversarial network (GAN). Generative adversarial networks (GANs) have achieved good results in image generation, style transfer, super-resolution reconstruction and other fields. In spatiotemporal fusion, generative adversarial networks (GANs) have also been applied and have shown good performance.

[0041] In the remote sensing image spatiotemporal fusion model DFTNet of the present invention, the generator has three convolution branches, each of which is cascaded by multiple convolution blocks in sequence to extract features, and the three convolution branches contain the same number of convolution blocks. By stacking convolution blocks, features of different scales are extracted in a local to global manner.

[0042] In this embodiment, each convolution block in each convolution branch includes a 3×3 convolution layer, a batch normalization layer, and a ReLU activation function for feature extraction. In each convolution block, the 3×3 convolution layer can filter and extract features from the input feature map, extract features in a local-to-global manner, and fully extract spatiotemporal change information and ground details; the batch normalization layer helps to accelerate the training process, reduce the problem of gradient disappearance or explosion, and improve the generalization ability of the remote sensing image spatiotemporal fusion model DFTNet; the ReLU activation function can set the negative part to zero, enhancing the expression ability of the remote sensing image spatiotemporal fusion model DFTNet.

[0043] In addition, in this embodiment, the stages of the remote sensing image spatiotemporal fusion model DFTNet are selected as two stages, that is, the convolution branch is composed of two convolution blocks cascaded in sequence, so that the remote sensing image spatiotemporal fusion model DFTNet learns more complex features and patterns, increases the expression ability of the remote sensing image spatiotemporal fusion model DFTNet, and helps to better fit the data.

[0044] After the remote sensing image spatiotemporal fusion model DFTNet obtains feature maps of different scales through different stages, the present invention introduces a spectrum coordination module with a Transformer structure to process the feature maps of different scales in terms of amplitude and phase, and obtains feature maps of different stages again. In addition, the present invention introduces a spatial information branch to extract information in the spatial domain from the fine image F1 of the input reference date. Then, the feature maps obtained by the spectrum coordination module at each stage and the feature maps obtained by the spatial information branch are spliced ​​in the dimension of the number of channels to avoid information loss. Compared with simply stacking convolutional layers, the feature information of each stage can be better retained, thereby obtaining a richer information feature representation. Finally, the spliced ​​feature map is processed with a standard 1×1 convolution layer to map its feature channel number to the number of channels of the real fine image, thereby generating a predicted image.

[0045] It should be noted that in the present invention, the remote sensing image spatiotemporal fusion model DFTNet needs to jointly update the generator and discriminator parameters during training, but during inference, only the trained generator needs to be used. The inputs in the training and inference stages are the coarse image C1 and fine image F1 of the reference date and the coarse image C2 of the predicted date to generate a high-resolution fine image F2′ (predicted image) of the predicted date.

[0046] Specifically, when training the remote sensing image spatiotemporal fusion model DFTNet, the above generator takes a pair of coarse images of the reference date, a fine image and a coarse image of the predicted date as input, performs multiple convolutions on each image input, and passes the feature maps of each stage obtained by the convolution into the spectrum coordination module. At the same time, the fine image F1 is passed into the spatial information branch. Finally, the output of the spectrum coordination module at each stage is spliced ​​with the spatial information feature map obtained by the spatial information branch. The spliced ​​fusion feature map is then convolved through 1×1 to map the number of feature channels to the number of channels of the real fine image F2, generating the fine image F2′ of the predicted date. The discriminator takes the fine image F2′ of the predicted date and the real fine image F2 as input, and makes a distinction through a binary convolutional neural network. If the output result of the discriminator is True, it means that the fine image F2′ of the predicted date is close to the real fine image F2. Conversely, if the output result of the discriminator is False, it means that the fine image F2′ of the predicted date is far away from the real fine image F2. The predicted image obtained by the remote sensing image spatiotemporal fusion model DFTNet can be infinitely close to the real image, which constitutes a training process.

[0047] It should also be noted that in the present invention, the coarse image and the fine image are each obtained by a sensor, and the sampling points corresponding to the two images are different, so the resolutions are different. Some dates can obtain both coarse images and fine images, while other dates can only obtain coarse images but no fine images. In the description of the present invention, the date on which the coarse image and the fine image can be obtained at the same time is called the reference date, and the date on which only the coarse image can be obtained but no fine image is called the predicted date.

[0048] It should be noted that, in the present invention, the spectrum coordination module processing flow in each stage is the same. In the spectrum coordination module of the i-th stage, Figure 1 As shown, the first intermediate feature map output by the i-th stage convolution block of the first convolution branch is obtained. The second intermediate feature map output by the i-th stage convolution block of the second convolution branch And the third intermediate feature map output by the i-th stage convolution block of the third convolution branch For the first intermediate feature map Perform discrete Fourier transform to obtain the first amplitude feature map and the first phase feature map, and the second intermediate feature map Perform discrete Fourier transform to obtain the second amplitude feature map and the second phase feature map, and the third intermediate feature map Perform discrete Fourier transform to obtain a third amplitude feature map and a third phase feature map, input the first amplitude feature map and the third amplitude feature map into the amplitude-based Transformer module, reconstruct the corrected amplitude, and obtain a new amplitude feature map, input the first phase feature map and the second phase feature map into the phase-based Transformer module, reconstruct the corrected phase, and obtain a new phase feature map, perform inverse discrete Fourier transform on the new phase feature map and the new amplitude feature map, and obtain the feature map output by the spectrum coordination module of the i-th stage.

[0049] The specific structure and principle of the above-mentioned spectrum coordination module are described in detail below. In the spectrum coordination module, the present invention introduces a Transformer model suitable for image reconstruction tasks, which is intended to process data composed of amplitude or phase obtained by Fourier transform, and based on this, an amplitude-based Transformer module and a phase-based Transformer module are constructed. In the amplitude-based Transformer module, it receives the amplitude information of the coarse image (C1) and the fine image (F1) corresponding to the reference date, that is, the above-mentioned first amplitude feature map and the third amplitude feature map. In the phase-based Transformer module, it receives the phase information of the coarse image (C1) corresponding to the reference date and the coarse image (C2) of the predicted date, that is, the above-mentioned first phase feature map and the second phase feature map. By processing the amplitude and phase information through the multi-head self-attention mechanism and feedforward network in the original Transformer model, feature interaction and integration at different levels and scales can be achieved. Among them, the multi-head self-attention mechanism helps to capture the global dependencies in the input information, enabling the remote sensing image spatiotemporal fusion model DFTNet to focus on the feature representations of different parts at the same time, while the feedforward network can perform nonlinear transformation and extraction on the features of each position, thereby enhancing the representation ability of the remote sensing image spatiotemporal fusion model DFTNet.

[0050] It should be noted that in the phase-based Transformer module or the amplitude-based Transformer module of the present invention, the first input data X and the second input data Y are calculated by the multidimensional convolution head transposition attention mechanism, and are projected by the key (Key, K), value (Value, V) and query (Query, Q) so that the relationship between different elements can be considered simultaneously when calculating the attention weight. These relationships are modeled by the multidimensional convolution head transposition attention mechanism, and each head captures different aspects of the input sequence, and finally a self-attention feature map is obtained. The self-attention feature map is passed through a gated multidimensional convolutional feedforward network to obtain a phase feature map or an amplitude feature map Z.

[0051] It should be noted that in the present invention, for the phase-based Transformer module, it is necessary to rely on the information of the coarse image C1 of the reference date to guide the generation of the coarse image C2 of the predicted date, so the first input data is the intermediate feature map output by the convolution block of each stage of the first convolution branch, and the second input data is the intermediate feature map output by the convolution block of each stage of the second convolution branch, that is, the first input data X corresponds to the coarse image C1 of the reference date, and the second input data Y corresponds to the coarse image C2 of the predicted date. In the amplitude-based Transformer module, it is necessary to rely on the information of the coarse image C1 of the reference date to guide the generation of the fine image F1 of the reference date, so the first input data is the intermediate feature map output by the convolution block of each stage of the first convolution branch, and the second input data is the intermediate feature map output by the convolution block of each stage of the third convolution branch, that is, the first input data X corresponds to the coarse image C1 of the reference date, and the second input data Y corresponds to the fine image F1 of the reference date.

[0052] In this embodiment, the intermediate processes of the amplitude-based Transformer module / phase-based Transformer module and the dimensions of each parameter are further described so that those skilled in the art can better understand the principle of using the amplitude-based Transformer module / phase-based Transformer module.

[0053] Specifically, the structure of the amplitude-based Transformer module / phase-based Transformer module is as follows Figure 2 As shown in FIG. 1 , it includes a multi-dimensional convolution head transposed attention (Multi-Dconv Head Transposed Attention, MDTA) mechanism and a gated multi-dimensional convolution feed-forward network (Gated-Dconv Feed-Forward Network, GDFN). The multi-head self-attention mechanism in the original Transformer model has a large amount of calculation and is not suitable for application to high-resolution images. Therefore, the present invention improves the multi-head self-attention mechanism into a multi-dimensional convolution head transposed attention mechanism, which is characterized by calculating the self-attention on the channel to implicitly encode the global context information.

[0054] The multi-dimensional convolution head transposition attention mechanism is a process of enriching and extracting information, which helps to more effectively capture long-distance dependencies in high-resolution images and improve the perception of global information by the remote sensing image spatiotemporal fusion model DFTNet. At the same time, the multi-dimensional convolution head transposition attention mechanism can also effectively reduce the computational complexity, making it more scalable when processing large-scale data. In addition, the multi-dimensional convolution head transposition attention mechanism uses depth-wise convolution operations to generate value features, key features, and query features when calculating the self-attention map, which helps to emphasize local information.

[0055] In the multi-dimensional convolutional head transposed attention mechanism, the first input data X is first normalized by LN(·), and the normalized first input data LN(X) is sequentially passed through the first convolutional layer and the first depth-wise convolutional layer to obtain the value feature In the formula, represents the weight matrix of the first depth-wise convolutional layer, Represents the weight matrix of the first convolutional layer. The normalized first input data passes through the second convolutional layer and the second depth-wise convolutional layer in turn to obtain the key feature In the formula, represents the weight matrix of the second depth-wise convolutional layer, Represents the weight matrix of the second convolutional layer. The second input data Y is normalized by the layer, and the normalized second input data LN(Y) is passed through the third convolutional layer and the third depth-wise convolutional layer in sequence to obtain the query feature In the formula, represents the weight matrix of the third depth-wise convolutional layer, Represents the weight matrix of the third convolutional layer.

[0056] The query feature Key Features And the value feature The attention is calculated as follows to obtain the transposed attention map:

[0057]

[0058] Where α is a learnable scaling parameter applied to the Softmax function, which is used to obtain the transposed attention map.

[0059] The transposed attention map is passed through the fourth convolutional layer to obtain a new transposed attention map, and the new transposed attention map is residually connected with the second input data Y to obtain a self-attention feature map.

[0060]

[0061] Where W p Represents the weight matrix of the fourth convolutional layer.

[0062] In this embodiment, the convolution kernel sizes of the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer are all 1×1, and the convolution kernel sizes of the first depth-wise convolution layer, the second depth-wise convolution layer and the third depth-wise convolution layer are all 3×3. The depth-wise convolution layer performs a convolution operation on each input channel with the convolution kernel separately, instead of combining all input channels together like the traditional convolution, which can reduce the number of parameters and the amount of calculation.

[0063] The gated multidimensional convolutional feedforward network (GDFN) introduces a gating mechanism and depth-wise convolution. Compared with the classic Transformer model that uses a feedforward network to perform pixel-by-pixel feature operations, the gated multidimensional convolutional feedforward network uses a gating unit to obtain the activation state of each pixel, uses a 1×1 convolutional layer to expand the number of channels, and then uses a 3×3 depth-wise convolutional layer and a GELU activation function to generate a gated map. The gated multidimensional convolutional feedforward network can be seen as a filter for information, which can control the flow of information in the forward propagation of the network.

[0064] The gated multidimensional convolutional feedforward network GDFN allows each layer to focus on subtle details that complement other layers by controlling the flow of information, which helps to filter and aggregate important feature information. This information screening process can help the remote sensing image spatiotemporal fusion model DFTNet learn and utilize global context information more effectively, thereby enhancing the remote sensing image spatiotemporal fusion model DFTNet's ability to represent and generalize input data.

[0065] In the gated multidimensional convolutional feedforward network, the self-attention feature map After layer normalization, the normalized self-attention feature map After the fifth convolutional layer, the fourth depth-wise convolutional layer and the GELU activation function, a gated map is generated. In the formula, represents the weight matrix of the fifth convolutional layer, represents the weight matrix of the fourth depth-wise convolutional layer, Represents the GELU nonlinear function. The normalized self-attention feature map is passed through the sixth convolutional layer and the fifth depth-wise convolutional layer in turn to obtain the gated feature map In the formula, represents the weight matrix of the sixth convolutional layer, Represents the weight matrix of the fifth depth-wise convolutional layer. Multiply the gated map and the gated feature map element by element to obtain the weighted gated feature map

[0066]

[0067] The weighted gated feature map is passed through the seventh convolutional layer to obtain a deep gated feature map. In the formula, Represents the weight matrix of the seventh convolutional layer. The depth gated feature map is combined with the self-attention feature map Perform residual connection to obtain the output feature map Z and use it as a phase feature map or an amplitude feature map:

[0068]

[0069] In this embodiment, the convolution kernel sizes of the fifth convolution layer, the sixth convolution layer, and the seventh convolution layer are all 1×1, and the convolution kernel sizes of the fourth depth-wise convolution layer and the fifth depth-wise convolution layer are all 3×3.

[0070] It should be noted that, in the present invention, the spatial information branch is formed by sequentially cascading the eighth convolutional layer, LRelu and the ninth convolutional layer, and the convolution kernel sizes of the eighth convolutional layer and the ninth convolutional layer are both 3×3.

[0071] In this embodiment, the spatial information branch includes a 3×3 convolution layer, LRelu and another 3×3 convolution layer in sequence. Its input is the fine image F1 of the reference date, and after being processed by the spatial information branch, a spatial information feature map output by the spatial information branch is obtained.

[0072] It should be noted that the specific processing flow in the discriminator of the present invention is as follows: the fine image F2′ generated by the generator and the true fine image F2 are first passed through a tenth convolutional layer and a LeakyReLU activation function to obtain a first feature map, the first feature map is processed in turn through four first convolutional blocks to obtain a second feature map, and finally the second feature map is passed through a twelfth convolutional layer and a sigmoid activation function to obtain the output result of the discriminator, wherein each first convolutional block includes an eleventh convolutional layer, batch normalization and a LeakyReLU activation function, the convolution kernel size of the tenth convolutional layer is 4×4, the convolution kernel size of the eleventh convolutional layer is 4×4, and the convolution kernel size of the twelfth convolutional layer is 4×4.

[0073] In this embodiment, the output result of the discriminator is a value of 0 or 1, where 0 indicates that the image is a predicted image and 1 indicates that the image is a real image.

[0074] In summary, in terms of significant resolution differences, images of different resolutions have differences in spectral characteristics due to different sensors. The present invention designs an amplitude-based Transformer module to reveal the differences in spectral characteristics and reconstruct the corrected amplitude. In terms of temporal feature differences, slight changes in the surface can be clearly presented in the phase information. The present invention designs a phase-based Transformer module to more accurately reveal changes in the structure of the ground objects and improve the accuracy of the analysis of time series data. In addition, due to the global analysis characteristics of the frequency domain, the present invention also designs a spatial information branch to make up for the processing of local feature texture information and enhance the perception of local details by the remote sensing image spatiotemporal fusion model. Therefore, the present invention provides an effective method to integrate Fourier transformed data, making it suitable for a variety of application scenarios that require understanding of frequency domain characteristics. Through in-depth experimental verification, the remote sensing image spatiotemporal fusion model DFTNet proposed in the present invention demonstrates superior performance in processing Fourier transform data, especially in capturing the complex relationship between amplitude and phase information.

[0075] The present invention will now use a specific example to demonstrate the application effect of the remote sensing image spatiotemporal fusion method based on the deep Fourier Transformer network described in the above embodiment on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0076] Example

[0077] The specific implementation process of the remote sensing image spatiotemporal fusion method based on the deep Fourier Transformer network used in this embodiment is as described above and will not be repeated here.

[0078] The overall process in this embodiment can be divided into three stages: data preprocessing, model training, and image generation. Figure 3 shown.

[0079] 1. Data preprocessing

[0080] The datasets used are CIA (the Coleambally Irrigation Area) and LGC (the lowerGwydir catchments (LGC)) open source datasets. These two areas represent phenology and land cover changes, respectively. The CIA dataset contains 17 pairs of cloud-free MODIS-Landsat images captured during 2001 and 2002; the LGC dataset contains 14 cloud-free image pairs captured during 2004 and 2005. The image sizes range from 1280×1792 to 3072×2056, and only the first four channels (i.e., B, G, R, and NIR) of all images are concerned.

[0081] 2. Model training

[0082] Step 1: Build a training dataset and divide it into batches according to a fixed batch size. Combine two pairs of data that are adjacent in time on this dataset into one set of data. The last four sets of data in each dataset are used as validation sets, and the rest are used as training sets.

[0083] Step 2: Use each batch of training samples to train the remote sensing image spatiotemporal fusion model (DFTNet). The specific structure of DFTNet is as described above and will not be repeated here. Figure 3 As shown in the figure, during the training process, the network parameters of the remote sensing image spatiotemporal fusion model need to be adjusted through the generator and the discriminator at the same time until all batches of the training data set participate in the remote sensing image spatiotemporal fusion model training. After reaching the specified number of iterations, the remote sensing image spatiotemporal fusion model converges and the training is completed.

[0084] 3. Image Generation

[0085] The images in the test set are directly used as input into the trained remote sensing image spatiotemporal fusion model DFTNet generator, and finally a high-resolution image under the predicted date is generated, thus realizing spatiotemporal fusion.

[0086] In this embodiment, the test results Figure 4 As shown. It can be seen that the remote sensing image spatiotemporal fusion model DFTNet of the present invention can generate predicted images well. In terms of visual effects, EDCSTFN and GAN-STFM show their limitations in details such as edges and textures when reconstructing images, resulting in the transformation of real scenes being blurred in the generated images. They still have room for improvement in capturing high-frequency details of images, especially in terms of edge sharpness and texture clarity. The remote sensing image spatiotemporal fusion model DFTNet has made significant progress in this regard, and it can show higher accuracy and clarity in the generation of details. This advantage makes the remote sensing image spatiotemporal fusion model DFTNet particularly effective in image reconstruction tasks with rich details or high contrast.

[0087] In the processing of spectral information, the remote sensing image spatiotemporal fusion model DFTNet successfully captured rich spectral information through in-depth research in the frequency domain. Compared with the performance of MS-Fusion in color restoration, DFTNet makes the acquisition and utilization of spectral information more accurate through frequency domain analysis, thus achieving better results in color authenticity and spectral fidelity. This makes the remote sensing image spatiotemporal fusion model DFTNet particularly important in fields such as remote sensing image analysis that require high spectral fidelity.

[0088] In terms of reproducing spatial details, the remote sensing image spatiotemporal fusion model DFTNet also showed excellent performance. It has a more obvious advantage than SwinSTFM and MLFF-GAN in restoring the spatial accuracy of images. This capability of the remote sensing image spatiotemporal fusion model DFTNet is particularly valuable for application scenarios that require precise detail resolution, such as fine terrain mapping, urban planning, and precision agricultural monitoring.

[0089] In the comparison between the remote sensing image spatiotemporal fusion model DFTNet and G-TFFN, careful observation shows that the quality of the images they generate is not significantly different. However, the remote sensing image spatiotemporal fusion model DFTNet has shown some progress in dealing with noise. This improvement may be due to the fact that the remote sensing image spatiotemporal fusion model DFTNet is easier to deal with noise in the frequency domain, making it more effective in separating noise components when extracting image features.

[0090] It should also be noted that the remote sensing image spatiotemporal fusion method based on the deep Fourier Transformer network in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a remote sensing image spatiotemporal fusion system based on a deep Fourier Transformer network corresponding to the remote sensing image spatiotemporal fusion method based on the deep Fourier Transformer network provided in the above embodiment, such as Figure 5 As shown, it includes:

[0091] A data acquisition module, used to acquire a coarse image of a reference date, a fine image of a reference date, and a coarse image of a predicted date;

[0092] The result acquisition module is used to input the coarse image of the reference date, the fine image of the reference date and the coarse image of the predicted date into the trained generator, and output the fine image of the predicted date; when training the generator, the generator and a discriminator are used to form a remote sensing image spatiotemporal fusion model, and the parameters of the remote sensing image spatiotemporal fusion model are jointly optimized, the generator is used to obtain the fine image of the predicted date, and the discriminator is used to distinguish the fine image generated by the generator from the real fine image, wherein the generator includes a spectrum coordination module and a spatial information branch, the spectrum coordination module includes an amplitude-based Transformer module and a phase-based Transformer module; the processing flow in the phase-based Transformer module and the amplitude-based Transformer module is the same, and the phase-based Transformer module or the amplitude-based Transformer module both include a multi-dimensional convolution head transposition attention mechanism and a gated multi-dimensional convolution feedforward network;

[0093] In the generator, the coarse image of the reference date is subjected to multi-stage convolution through the first convolution branch to extract features, the coarse image of the predicted date is subjected to multi-stage convolution through the second convolution branch to extract features, and the fine image of the reference date is subjected to multi-stage convolution through the third convolution branch to extract features. Each convolution branch is formed by sequentially cascading a plurality of convolution blocks whose parameters are not shared. The first convolution branch, the second convolution branch and the third convolution branch contain the same number of convolution blocks. The convolution blocks of each stage in each convolution branch output an intermediate feature map. In the three convolution branches, the intermediate feature maps output by the convolution blocks of the same stage are used as the input of the spectrum coordination module of the corresponding stage, and the fine image of the reference date is passed into the spatial information branch. Then, the feature maps output by the spectrum coordination modules of each stage and the spatial information feature maps output by the spatial information branch are spliced ​​to obtain a fused feature map. The fused feature map is passed through a standard convolution layer to obtain a fine image of the predicted date.

[0094] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A remote sensing image spatiotemporal fusion method based on a deep Fourier Transformer network, characterized in that: The following steps are involved: Input the coarse image and fine image of the reference date and the coarse image of the predicted date into the trained generator, and output the fine image of the predicted date; when training the generator, the generator and a discriminator constitute a remote sensing image spatiotemporal fusion model, and jointly optimize the parameters of the remote sensing image spatiotemporal fusion model, the generator is used to obtain the fine image of the predicted date, and the discriminator is used to distinguish the fine image generated by the generator from the real fine image, wherein the generator includes a spectrum coordination module and a spatial information branch, the spectrum coordination module includes an amplitude-based Transformer module and a phase-based Transformer module; the processing flow in the phase-based Transformer module and the amplitude-based Transformer module is the same, and the phase-based Transformer module or the amplitude-based Transformer module both include a multi-dimensional convolution head transposition attention mechanism and a gated multi-dimensional convolution feedforward network; In the generator, the coarse image of the reference date is subjected to multi-stage convolution through the first convolution branch to extract features, the coarse image of the predicted date is subjected to multi-stage convolution through the second convolution branch to extract features, and the fine image of the reference date is subjected to multi-stage convolution through the third convolution branch to extract features. Each convolution branch is formed by sequentially cascading a plurality of convolution blocks whose parameters are not shared. The first convolution branch, the second convolution branch and the third convolution branch contain the same number of convolution blocks. The convolution blocks of each stage in each convolution branch output an intermediate feature map. In the three convolution branches, the intermediate feature maps output by the convolution blocks of the same stage are used as the input of the spectrum coordination module of the corresponding stage, and the fine image of the reference date is passed into the spatial information branch. Then, the feature maps output by the spectrum coordination modules of each stage and the spatial information feature maps output by the spatial information branch are spliced ​​to obtain a fused feature map. The fused feature map is passed through a standard convolution layer to obtain a fine image of the predicted date.

2. The remote sensing image spatiotemporal fusion method based on the deep Fourier Transformer network according to claim 1, characterized in that: In each convolution branch of the generator, the convolution block at each stage contains a 3×3 convolution layer, a batch normalization layer, and a ReLU activation function.

3. The remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network according to claim 1, characterized in that: The processing flow of the spectrum coordination module in each stage is the same. In the spectrum coordination module of the i-th stage, the first intermediate feature map output by the i-th stage convolution block of the first convolution branch, the second intermediate feature map output by the i-th stage convolution block of the second convolution branch, and the third intermediate feature map output by the i-th stage convolution block of the third convolution branch are obtained, and the first intermediate feature map is discrete Fourier transformed to obtain a first amplitude feature map and a first phase feature map. The second intermediate feature map is discrete Fourier transformed to obtain a second amplitude feature map and a second phase feature map. The third intermediate feature map is discrete Fourier transformed to obtain a third amplitude feature map and a third phase feature map. The first amplitude feature map and the third amplitude feature map are input into the amplitude-based Transformer module to reconstruct the corrected amplitude to obtain a new amplitude feature map. The first phase feature map and the second phase feature map are input into the phase-based Transformer module to reconstruct the corrected phase to obtain a new phase feature map. The new phase feature map and the new amplitude feature map are inversely discrete Fourier transformed to obtain the feature map output by the i-th stage spectrum coordination module.

4. The remote sensing image spatiotemporal fusion method based on a deep Fourier Transformer network according to claim 1, characterized in that: In a phase-based Transformer module or an amplitude-based Transformer module, the first input data and the second input data are calculated by a multidimensional convolution head transposition attention mechanism to obtain a self-attention feature map, and the self-attention feature map is passed through a gated multidimensional convolution feedforward network to obtain a phase feature map or an amplitude feature map; In the phase-based Transformer module, the first input data is the intermediate feature map output by the convolution block of each stage of the first convolution branch, and the second input data is the intermediate feature map output by the convolution block of each stage of the second convolution branch; in the amplitude-based Transformer module, the first input data is the intermediate feature map output by the convolution block of each stage of the first convolution branch, and the second input data is the intermediate feature map output by the convolution block of each stage of the third convolution branch.

5. The remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network according to claim 4, characterized in that: In the transposed attention mechanism of the multi-dimensional convolutional head, the first input data is layer-normalized, and the normalized first input data is sequentially passed through the first convolution layer and the first depth-wise convolution layer to obtain the value feature, and the normalized first input data is sequentially passed through the second convolution layer and the second depth-wise convolution layer to obtain the key feature, and the second input data is layer-normalized, and the normalized second input data is sequentially passed through the third convolution layer and the third depth-wise convolution layer to obtain the query feature, and the query feature, key feature and value feature are attention-calculated to obtain the transposed attention map, and the transposed attention map is passed through the fourth convolution layer to obtain a new transposed attention map, and the new transposed attention map is residually connected with the second input data to obtain a self-attention feature map.

6. The remote sensing image spatiotemporal fusion method based on a deep Fourier Transformer network as claimed in claim 5, characterized in that: In the gated multidimensional convolutional feedforward network, the self-attention feature map is layer-normalized, and the normalized self-attention feature map is sequentially passed through the fifth convolution layer, the fourth depth-wise convolution layer and the GELU activation function to generate a gated mapping map. The normalized self-attention feature map is sequentially passed through the sixth convolution layer and the fifth depth-wise convolution layer to obtain a gated feature map. The gated mapping map and the gated feature map are element-wise multiplied to obtain a weighted gated feature map. The weighted gated feature map is passed through the seventh convolution layer to obtain a depth gated feature map. The depth gated feature map is residually connected with the self-attention feature map to obtain an output feature map and use it as a phase feature map or an amplitude feature map.

7. The remote sensing image spatiotemporal fusion method based on a deep Fourier Transformer network as claimed in claim 6, characterized in that: The convolution kernel sizes of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer, the sixth convolution layer, and the seventh convolution layer are all 1×1, and the convolution kernel sizes of the first depth-wise convolution layer, the second depth-wise convolution layer, the third depth-wise convolution layer, the fourth depth-wise convolution layer, and the fifth depth-wise convolution layer are all 3×3.

8. The remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network according to claim 1, characterized in that: The spatial information branch is composed of the eighth convolutional layer, LRelu and the ninth convolutional layer cascaded in sequence. The convolution kernel sizes of the eighth and ninth convolutional layers are both 3×3.

9. The remote sensing image spatiotemporal fusion method based on deep Fourier Transformer network according to claim 1, characterized in that: The specific processing flow of the discriminator is as follows: the fine image generated by the generator and the real fine image are first passed through a tenth convolution layer and a LeakyReLU activation function to obtain a first feature map, the first feature map is processed in turn through four first convolution blocks to obtain a second feature map, and finally the second feature map is passed through a twelfth convolution layer and a sigmoid activation function to obtain the output result of the discriminator, wherein each first convolution block includes an eleventh convolution layer, batch normalization and a LeakyReLU activation function, the convolution kernel size of the tenth convolution layer is 4×4, the convolution kernel size of the eleventh convolution layer is 4×4, and the convolution kernel size of the twelfth convolution layer is 4×4.

10. A remote sensing image spatiotemporal fusion system based on a deep Fourier Transformer network, characterized in that: include: A data acquisition module, used to acquire a coarse image of a reference date, a fine image of a reference date, and a coarse image of a predicted date; The result acquisition module is used to input the coarse image of the reference date, the fine image of the reference date and the coarse image of the predicted date into the trained generator, and output the fine image of the predicted date; when training the generator, the generator and a discriminator are used to form a remote sensing image spatiotemporal fusion model, and the parameters of the remote sensing image spatiotemporal fusion model are jointly optimized, the generator is used to obtain the fine image of the predicted date, and the discriminator is used to distinguish the fine image generated by the generator from the real fine image, wherein the generator includes a spectrum coordination module and a spatial information branch, the spectrum coordination module includes an amplitude-based Transformer module and a phase-based Transformer module; the processing flow in the phase-based Transformer module and the amplitude-based Transformer module is the same, and the phase-based Transformer module or the amplitude-based Transformer module both include a multi-dimensional convolution head transposition attention mechanism and a gated multi-dimensional convolution feedforward network; In the generator, the coarse image of the reference date is subjected to multi-stage convolution through the first convolution branch to extract features, the coarse image of the predicted date is subjected to multi-stage convolution through the second convolution branch to extract features, and the fine image of the reference date is subjected to multi-stage convolution through the third convolution branch to extract features. Each convolution branch is formed by sequentially cascading a plurality of convolution blocks whose parameters are not shared. The first convolution branch, the second convolution branch and the third convolution branch contain the same number of convolution blocks. The convolution blocks of each stage in each convolution branch output an intermediate feature map. In the three convolution branches, the intermediate feature maps output by the convolution blocks of the same stage are used as the input of the spectrum coordination module of the corresponding stage, and the fine image of the reference date is passed into the spatial information branch. Then, the feature maps output by the spectrum coordination modules of each stage and the spatial information feature maps output by the spatial information branch are spliced ​​to obtain a fused feature map. The fused feature map is passed through a standard convolution layer to obtain a fine image of the predicted date.

Citation Information

Patent Citations

  • Lightweight optical remote sensing image change detection method based on efficient attention

    CN115713529A

  • Remote sensing image target detection method based on fusion convolution attention mechanism

    CN116229295A

  • Remote sensing image space-time fusion method based on adaptive neural network

    CN116563103A

  • Remote sensing image change detection method based on time-space interaction Transform model

    CN117095287A

  • Double-branch remote sensing image space-time fusion method based on kernel attention selection

    CN117745556A

Cited By

  • Image fusion method and device, storage medium and program product

    CN120543395A

  • Remote sensing image land development category segmentation method and system based on edge detection

    CN120673064A

  • Land development category segmentation method and system based on edge detection of remote sensing image

    CN120673064B

  • Remote sensing image sharpening method based on frequency domain hybrid expert and spatial domain difference

    CN121032853A