A remote sensing image sharpening method based on frequency domain hybrid expert and space domain difference

Abstract

The present application relates to a kind of based on frequency domain hybrid expert and space domain difference remote sensing image sharpening method, belong to difference learning and remote sensing image processing technical field.Full color image and up-sampled low-resolution hyperspectral image are respectively frequency domain decoupling, obtain amplitude feature map and phase feature map;In frequency domain, amplitude feature map is learned using hybrid expert amplitude component learning module, and phase feature map is learned using hybrid expert phase component learning module;In spatial domain, space domain difference feature is extracted;After amplitude component learning feature and phase component learning feature are input into space-frequency domain feature fusion module by inverse fourier transform, fusion, after the feature and space domain difference feature are added, network output image is obtained, and high-resolution hyperspectral image is finally obtained by continuous iteration optimization.The present application method effectively sharpens low-resolution multispectral remote sensing image, to some extent, make up for the deficiency of the existing method based on deep learning in low-resolution multispectral remote sensing image sharpening task.

Description

A Remote Sensing Image Sharpening Method Based on Frequency Domain Hybridization Expert and Spatial Domain Difference Technical Field

[0001] This invention relates to a remote sensing image sharpening method based on frequency domain hybridization experts and spatial domain differences, belonging to the fields of difference learning and remote sensing image processing technology. Background Technology

[0002] With the development of remote sensing, remote sensing imagery plays a vital role in many fields, such as map services, precision agriculture, and urban development. However, due to hardware limitations, satellite sensors can only capture high-resolution single-channel panchromatic images and low-resolution hyperspectral images; they cannot capture high spatial resolution multispectral images. Low-resolution multispectral image sharpening is an important but challenging remote sensing image enhancement task, aiming to generate high spatial resolution multispectral images from coupled low spatial resolution multispectral and panchromatic images.

[0003] Existing low-resolution multispectral image sharpening methods can be broadly categorized into two types: traditional methods and deep learning-based methods. Typical traditional methods include component substitution, multiresolution analysis, and variational optimization. These methods rely heavily on subjective prior assumptions during the sharpening process. While they have a solid theoretical foundation, their sharpening quality in real-world scenarios is inferior to that generated by deep learning methods. Deep learning-based methods, on the other hand, benefit from the superior representational capabilities of their layered paradigm. They can directly learn dense prior knowledge in the feature space and exhibit powerful sharpening performance.

[0004] The task of low-resolution multispectral image sharpening aims to enhance the spatial details of low spatial resolution multispectral images, including high-frequency structural information and low-frequency background information, guided by panchromatic images. While existing deep learning-based methods have conducted some research in the frequency domain, considering the diverse land cover types inherent in remote sensing images, previous frequency-domain methods only employ a single set of learnable parameters to learn the image's frequency components. This limits the model's ability to represent multi-land cover remote sensing images. On the other hand, considering the significant differences between panchromatic and low-resolution hyperspectral images in the spatial domain, especially the different semantic information contained in each channel of the low-resolution hyperspectral image, effectively learning these spatial differences can guide the model to generate more refined sharpening results. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned shortcomings and provide a remote sensing image sharpening method based on frequency domain mixing experts and spatial domain differences, so as to achieve superior low-resolution multispectral image sharpening effect.

[0006] The technical solution adopted in this invention is as follows:

[0007] A remote sensing image sharpening method based on frequency domain hybridization expert and spatial domain difference includes the following steps:

[0008] S1. Acquire remote sensing images, construct triplet data including panchromatic images, low-resolution hyperspectral images, and ground truth images, preprocess and divide the dataset;

[0009] S2. The amplitude and phase components of the panchromatic image and the upsampled low-resolution hyperspectral image are obtained by Fourier transform to achieve frequency domain decoupling. The amplitude components of the two images are then concatenated and convolved to obtain amplitude feature maps, and the phase components of the two images are then concatenated and convolved to obtain phase feature maps.

[0010] S3. In the frequency domain, the amplitude feature map is learned using the hybrid expert amplitude component learning module, and the phase feature map is learned using the hybrid expert phase component learning module. Both the hybrid expert amplitude component learning module and the hybrid expert phase component learning module include an expert library and a calculated hybrid expert weight library. The output feature map of each expert is weighted and fused with its corresponding weight in the hybrid expert weight library to obtain the corresponding component learning feature map.

[0011] S4. In the spatial domain, the panchromatic image is subtracted from the upsampled low-resolution hyperspectral image band by band to extract the spectral difference information between the two. The final spatial domain difference features are obtained through convolution operation and ReLU activation function.

[0012] S5. The amplitude component learning feature map and the phase component learning feature map are inverse Fourier transformed to obtain the enhanced representation features of the low-resolution hyperspectral image in the frequency domain. The enhanced representation features in the frequency domain are then concatenated with the spatial difference features in the channel dimension and then input into the spatial-frequency domain feature fusion module for fusion.

[0013] S6. The fused features are added to the spatial difference features to obtain the network output image. Then, the loss between the network output image and the ground truth is calculated. After continuous iterative optimization, a high-resolution hyperspectral image is finally obtained.

[0014] S7. Train the model using the training set, and test the model using the test set in different multispectral scenarios to obtain the final model.

[0015] In the above method, the hybrid expert amplitude component learning module described in step S3 has an expert library containing N experts. Each expert is designed as a lightweight depthwise separable convolution with a kernel size of 1×1. The input amplitude feature map is processed in parallel using Global Average Pooling (GAP) and Global Max Pooling (GMP), and the results are summed. The aggregated feature vector is then fed into a fully connected layer (FC), whose output dimension is equal to the number of experts N. The output of this layer can be regarded as the activation score of each expert. The score vector is fed into the SoftMax function to generate the final expert weights. These weights together constitute the hybrid expert weight library. The output feature map of each expert is multiplied by its corresponding weight value in the hybrid expert weight library, and the weighted results are finally summed to obtain the amplitude component learning feature map.

[0016] The top K score vectors with the largest values ​​are selected first and fed into the SoftMax function to generate the final expert weights.

[0017] The structure of the hybrid expert phase component learning module is the same as that of the hybrid expert amplitude component learning module, except that it learns the phase feature map to obtain the phase component learning feature map.

[0018] The spatial-frequency domain feature fusion module described in step S5 consists of a 3×3 convolution, a 1×1 convolution, and a ReLU activation function, wherein the number of output channels of the 1×1 convolution is the number of input channels (i.e., the number of bands) of the low-resolution hyperspectral image.

[0019] Step S6 uses the mean absolute value loss function to supervise the training of the output feature map mapped back to the target dimension in the spatial-frequency domain feature fusion module and the ground-value high spatial resolution multispectral image, while using the squared coefficient of variation loss function to prevent uneven use of hybrid experts.

[0020] Another objective of this invention is to provide a remote sensing image sharpening device based on frequency domain mixing experts and spatial domain differences, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the remote sensing image sharpening method based on frequency domain mixing experts and spatial domain differences as described above.

[0021] A remote sensing image sharpening storage device based on frequency domain mixing experts and spatial domain differences stores a computer program, which, when executed, implements the steps of the remote sensing image sharpening method based on frequency domain mixing experts and spatial domain differences as described above.

[0022] The beneficial effects of this invention are:

[0023] (1) In the frequency domain, Fourier transform, advanced hybrid expert structure, and physical characteristics of remote sensing images are closely integrated. Hybrid experts are used to learn the decoupled frequency domain components. Compared with the traditional "one-size-fits-all" single network, this method can intelligently call the most suitable expert combination for processing according to the different input ground features. This enables the model to enhance the spatial resolution of the image while preserving the original spectral information to the greatest extent, effectively suppressing artifacts and spectral distortion, and the output results are more faithful to the real features of the ground features. The application of the Top-K mechanism is not only an innovation in model structure, but also a significant improvement in computational efficiency and model generalization ability.

[0024] (2) For the spatial domain, the method of the present invention can fully explore the semantic difference features between panchromatic images and low-resolution hyperspectral images, and further realize the subsequent spatial-frequency dual-domain feature complementarity, providing a novel and efficient technical path for the low-resolution hyperspectral image sharpening task.

[0025] (3) This invention effectively sharpens low-resolution multispectral remote sensing images by combining hybrid experts in decoupled frequency domain component learning and difference learning based on image spatial domain, which to some extent makes up for the shortcomings of existing deep learning-based methods in the task of sharpening low-resolution multispectral remote sensing images. Attached Figure Description

[0026] Figure 1 is a flowchart of the method of the present invention;

[0027] Figure 2 is a model architecture diagram of the method of the present invention. Detailed Implementation

[0028] The following description, in conjunction with specific embodiments, provides further details.

[0029] Example 1: A remote sensing image sharpening method based on frequency domain hybridization experts and spatial domain differences, comprising the following steps:

[0030] S1. Acquire remote sensing images, construct triplet data including panchromatic images, low-resolution hyperspectral images, and ground truth images, preprocess and partition the dataset:

[0031] To ensure the rigor of the experimental data for this low-resolution hyperspectral image sharpening task, the method of this invention uses the internationally recognized PanCollection dataset, whose data format strictly adheres to the Wald protocol. This dataset contains remote sensing images captured by World-View3 (8 bands, 11 bits), GaoFen2 (4 bands, 10 bits), and QuickBird (4 bands, 11 bits) satellites. Specifically, the dataset is divided into training and validation sets in a 9:1 ratio. Each set consists of a triplet: panchromatic image / low-resolution hyperspectral image / ground ground truth image, with dimensions of 64×64×1 / 16×16×C / 64×64×C, where C is the number of image bands. The test set uses 20 triplet images, with dimensions of 256×256×1 / 64×64×C / 256×256×C. We perform intensity upper bound reduction on the pixel intensity of all samples in the loaded dataset to accelerate the convergence speed of the proposed image augmentation model during training. Specifically, for 11-bit remote sensing images, we normalize by dividing the data by 2047, and for 10-bit remote sensing images, we normalize by dividing the data by 1023.

[0032] S2. Fourier transform is applied to the panchromatic image and the upsampled low-resolution hyperspectral image to obtain the corresponding amplitude and phase components, achieving frequency domain decoupling. Channel concatenation and convolution are then performed on the amplitude components of the two images to obtain amplitude feature maps, and channel concatenation and convolution are performed on the phase components of the two images to obtain phase feature maps.

[0033] As shown in Figure 2, since the spatial resolution of the low spatial resolution multispectral image captured by the satellite sensor is 4 times different from that of the panchromatic image, the low spatial resolution multispectral image of the training set is first upsampled to 64×64×C by pixel rearrangement, and then the panchromatic image and the upsampled low spatial resolution multispectral image are decoupled in the frequency domain.

[0034] For frequency domain decoupling, we employ a two-dimensional discrete Fourier transform to convert the panchromatic image and the upsampled multispectral image from the spatial domain to their complex components in the Fourier frequency domain. Then, we calculate the corresponding real and imaginary parts based on these complex components, and further calculate the corresponding amplitude and phase components to achieve frequency domain decoupling. The panchromatic image has one component channel, and the multispectral image has its number of bands C. To enable efficient learning of the decoupled frequency domain components by the hybrid expert, we concatenate the amplitude and phase components corresponding to the panchromatic and multispectral images separately, and then map the component features to a high-dimensional space using a convolution operation with a kernel size of 1 for subsequent learning.

[0035] S3. In the frequency domain, the amplitude feature map is learned using the hybrid expert amplitude component learning module, and the phase feature map is learned using the hybrid expert phase component learning module. Both the hybrid expert amplitude component learning module and the hybrid expert phase component learning module include an expert library and a calculated hybrid expert weight library. The output feature map of each expert is weighted and fused with its corresponding weight in the hybrid expert weight library to obtain the corresponding component learning feature map.

[0036] In the frequency domain, hybrid experts with the same structure are used to learn the amplitude and phase components separately. Faced with remote sensing images with a rich variety of ground features, the hybrid experts can adaptively activate their appropriate experts based on the specific low-frequency background information of the amplitude and the high-frequency structural information of the phase in the input image, aiming to maximize the learning of the most faithful image features.

[0037] There are two structurally identical but parameter-independent sub-modules: a hybrid expert amplitude component learning module and a hybrid expert phase component learning module. Taking the hybrid expert amplitude component learning module as an example, its core objective is to adaptively activate the corresponding spectral fidelity expert or content context expert based on the low-frequency background information and overall spectral distribution of the input image. Its detailed processing flow is as follows:

[0038] Given an amplitude feature map A obtained by concatenating channels and performing a 1×1 convolution, the calculation is as follows:

[0039] ,

[0040] Hybrid Expert Database: We predefine an expert database containing N experts. Each expert is designed as a lightweight depthwise separable convolution with a kernel size of 1×1. This design efficiently performs feature transformation and information extraction between channels while maintaining low computational complexity.

[0041] Hybrid Expert Weight Library: We apply Global Average Pooling (GAP) and Global Max Pooling (GMP) in parallel to the input amplitude feature map A, and sum the results to obtain a compact vector that represents the global context and the most salient features. The aggregated feature vector is then fed into a fully connected layer (FC) to obtain F. AThe output dimension equals the number of experts, N. The output of this layer can be considered as the activation score of each expert. To improve the efficiency and specialization of the model, we introduce a sparse gating strategy. From the N activation scores, we select the top K largest values, and set the remaining NK scores to negative infinity. This ensures that only a few of the most relevant experts are activated at any given time, avoiding resource waste and encouraging each expert to learn more discriminative features. Finally, the Top-K processed score vector is fed into the SoftMax function to generate the final expert weights. At this point, only the selected K experts have weights greater than zero, and the rest have weights of zero. The sum of all weights is 1. These weights together constitute the hybrid expert weight library, the internal process of which is as follows:

[0042] ,

[0043] ,

[0044] ,

[0045] F i A To activate the i-th score in the score set, where i ∈ 1, 2, ..., N. W A The amplitude-mixed expert weight library has a dimension of the number of experts N, corresponding to the weight of each expert.

[0046] Weighted fusion: The input amplitude feature map is fed in parallel to all N experts in the expert database. Then, the output feature map of each expert is multiplied by its corresponding weight value in the hybrid expert weight database. Finally, all weighted results are summed. Mathematically, the process can be represented as:

[0047] ,

[0048] Where E i A W is the i-th expert in the amplitude mixing expert database. i A Let A be the i-th weight in the amplitude expert weight library. ’ This is the amplitude component learning feature map obtained after hybrid expert amplitude component learning.

[0049] The hybrid expert phase component learning module employs the exact same structure, but it is trained on independent phase features P. Its goal is to learn how to activate corresponding texture or edge experts based on the structural information of the input image, thereby accurately enhancing and reconstructing high-frequency details, resulting in a phase component learning feature map P obtained after hybrid expert amplitude component learning. ’ .

[0050] S4. Subtract the panchromatic image from the upsampled low-resolution hyperspectral image band by band to extract the spectral difference information between the two. Then, perform convolution and ReLU activation to obtain the final spatial difference features:

[0051] For spatial difference learning, we subtract the panchromatic image from the upsampled low-resolution hyperspectral image band by band to explicitly extract the spectral difference information between the two. That is, for each band of the hyperspectral image, we calculate the difference features with the single-band panchromatic image. These difference features reveal the semantic information that the low-resolution hyperspectral image needs more attention in the spatial domain. We use a convolutional kernel of size 3 to mine deeper difference representations, and then use the ReLU activation function to obtain the final spatial difference features. The process is as follows:

[0052] ,

[0053] ,

[0054] The purpose of the duplicate operation is to copy the number of channels of the panchromatic image (PAN) to the same number of channels as the MS image, so as to achieve subsequent band subtraction, which is equivalent to the idea of ​​band subtraction in Figure 2. Diff represents the final spatial difference feature.

[0055] S5. The amplitude component and phase component learned feature maps are inverse Fourier transformed to obtain the enhanced representation features of the low-resolution hyperspectral image in the frequency domain. The enhanced representation features in the frequency domain are then concatenated with the spatial domain difference features along the channel dimension, and then input into the spatial-frequency domain feature fusion module for fusion.

[0056] Before spatial-frequency domain feature fusion, we use two-dimensional inverse discrete Fourier transform to obtain the enhanced representation of the low-resolution hyperspectral image in the frequency domain from the amplitude component learned feature map A' and the phase component learned feature map P'. Then, we concatenate this with the difference feature Diff learned in the spatial domain along the channel dimension, and finally perform spatial-frequency domain feature fusion. The spatial-frequency domain feature fusion module consists of 3×3 convolutions, 1×1 convolutions, and a ReLU activation function, aiming to achieve complementary learning of spatial and frequency domain features. The output channel number of the 1×1 convolution is the input channel number C (i.e., the number of bands) of the low-resolution hyperspectral image, aiming to map the features back to the target dimension C.

[0057] S6. The fused features are added to the spatial difference features to obtain the network output image. Then, the loss between the network output image and the ground truth is calculated. After continuous iterative optimization, a high-resolution hyperspectral image is finally obtained.

[0058] S7. Train the model using the training set, and test the model using the test set in different multispectral scenarios to obtain the final model:

[0059] This invention uses the mean absolute value loss function L1 to supervise the training of the output feature map mapped back to the target dimension C in the spatial-frequency domain feature fusion module and the ground-value high spatial resolution multispectral image, while using the squared coefficient of variation loss function L... scv To prevent uneven use of mixed experts, the training setup involved approximately 360 epochs on the 8-band dataset and approximately 150 epochs on the 4-band dataset, using the Adam optimizer with parameters of 0.9 and 0.999. Additionally, a quadratic coefficient of variation loss function was multiplied by a hyperparameter of 0.3, with an initial learning rate of 0.001, and a descent factor of 0.5 was applied every 100 epochs. The total number of amplitude and phase experts was 4, with 2 experts activated per epoch. During model training, the loss value corresponding to the training set was calculated after each iteration, and the model after the specified iterations was used as the final model. In the model testing phase, the model was tested on a test set in different multispectral scenarios. The model could adaptively sharpen low-spatial-resolution multispectral images to obtain high-spatial-resolution multispectral images.

[0060] The formula for calculating the absolute value loss function is as follows:

[0061] ,

[0062] Where Output is the network's output image, and GT is the ground truth image;

[0063] Formula for calculating the squared coefficient of variation loss function:

[0064] ,

[0065] Among them W A W is an amplitude expert weighting database. P As a phase expert weight library, SCV is responsible for calculating the mean along the batch dimension.

[0066] Example 2: Remote sensing image sharpening device based on frequency domain mixing expert and spatial domain difference, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the remote sensing image sharpening method based on frequency domain mixing expert and spatial domain difference as described in Example 1 above.

[0067] A remote sensing image sharpening storage device based on frequency domain mixing experts and spatial domain differences stores a computer program. When the program is executed, it implements the steps of the remote sensing image sharpening method based on frequency domain mixing experts and spatial domain differences as described in Embodiment 1 above.

[0068] The above is a further description of the present invention in conjunction with embodiments, and the scope of protection of the present invention is not limited thereto.

Claims

1. A remote sensing image sharpening method based on frequency domain hybridization experts and spatial domain differences, characterized in that, The steps are as follows: S1. Acquire remote sensing images, construct triplet data including panchromatic images, low-resolution hyperspectral images, and ground truth images, preprocess and divide the dataset; S2. Use Fourier transform to obtain the corresponding amplitude and phase components of the panchromatic image and the upsampled low-resolution hyperspectral image respectively to achieve frequency domain decoupling, and perform channel stitching and convolution operations on the amplitude components of the two images to obtain amplitude feature maps, and perform channel stitching and convolution operations on the phase components of the two images to obtain phase feature maps; S3. Use the hybrid expert amplitude component learning module to learn the amplitude feature maps in the frequency domain, and use the hybrid expert phase component learning module to learn the phase feature maps. Both the hybrid expert amplitude component learning module and the hybrid expert phase component learning module include an expert library and a calculated hybrid expert weight library. The output feature map of each expert is weighted and fused with its corresponding weight in the hybrid expert weight library to obtain the corresponding component learning feature map; S4. Subtract the panchromatic image from the upsampled low-resolution hyperspectral image band by band to extract the spectral difference information between the two, and obtain the final spatial difference features through convolution operation and ReLU activation function. S5. The amplitude component learning feature map and the phase component learning feature map are inverse Fourier transformed to obtain the enhanced representation features of the low-resolution hyperspectral image in the frequency domain. The enhanced representation features in the frequency domain are then concatenated with the spatial difference features in the channel dimension and then input into the spatial-frequency domain feature fusion module for fusion. S6. The fused features are added to the spatial difference features to obtain the network output image. Then, the loss between the network output image and the ground truth is calculated. After continuous iterative optimization, a high-resolution hyperspectral image is finally obtained. S7. Train the model using the training set, and test the model using the test set in different multispectral scenarios to obtain the final model.

2. The remote sensing image sharpening method based on frequency domain hybridization expert and spatial domain difference as described in claim 1, characterized in that, The hybrid expert amplitude component learning module described in step S3 has an expert library containing N experts. Each expert is designed as a lightweight depthwise separable convolution with a kernel size of 1×1. The input amplitude feature map is processed in parallel using Global Average Pooling (GAP) and Global Max Pooling (GMP), and the results are summed. The aggregated feature vector is then fed into a fully connected layer (FC), whose output dimension is equal to the number of experts N. The output of this layer can be regarded as the activation score of each expert. The score vector is fed into the SoftMax function to generate the final expert weights. These weights together constitute the hybrid expert weight library. The output feature map of each expert is multiplied by its corresponding weight value in the hybrid expert weight library, and the weighted results are finally summed to obtain the amplitude component learning feature map.

3. The remote sensing image sharpening method based on frequency domain hybridization expert and spatial domain difference according to claim 2, characterized in that, Step S3 selects the top K score vectors with the largest values ​​and feeds them into the SoftMax function to generate the final expert weights.

4. The remote sensing image sharpening method based on frequency domain hybridization expert and spatial domain difference as described in claim 1, characterized in that, The structure of the hybrid expert phase component learning module is the same as that of the hybrid expert amplitude component learning module, except that it learns the phase feature map to obtain the phase component learning feature map.

5. The remote sensing image sharpening method based on frequency domain hybridization expert and spatial domain difference according to claim 1, characterized in that, The spatial-frequency domain feature fusion module described in step S5 consists of a 3×3 convolution, a 1×1 convolution, and a ReLU activation function, wherein the number of output channels of the 1×1 convolution is the number of input channels of the low-resolution hyperspectral image.

6. A remote sensing image sharpening method based on frequency domain hybridization expert and spatial domain difference according to claim 1, characterized in that, Step S6 uses the mean absolute value loss function to supervise the training of the output feature map mapped back to the target dimension in the spatial-frequency domain feature fusion module and the ground-value high spatial resolution multispectral image, while using the squared coefficient of variation loss function to prevent uneven use of hybrid experts.

7. A remote sensing image sharpening device based on frequency domain hybridization and spatial domain difference, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the remote sensing image sharpening method based on frequency domain mixing experts and spatial domain differences as described in any one of claims 1-6.

8. A remote sensing image sharpening and storage device based on frequency domain hybridization experts and spatial domain differences, wherein a computer program is stored thereon, characterized in that, When the program is executed, the steps of the remote sensing image sharpening method based on frequency domain mixing experts and spatial domain differences as described in any one of claims 1-6 are implemented.

Images