A remote sensing image panchromatic sharpening method based on dual-domain collaborative learning

Abstract

The application discloses a kind of based on double-domain collaborative learning's remote sensing image panchromatic sharpening method, it is related to image processing field.In spatial domain, local details of source image are extracted using CNN, global context is modeled combining Mamba model, and deep fusion is carried out through attention mechanism;In frequency domain, the image is decoupled using Fourier transform, and on the basis of strictly maintaining the phase spectrum of dominant structure, the two modal amplitude spectrum is dynamically weighted and fused;Subsequently, through the space-frequency interaction module and the adaptive cross-domain complementary rule, redundant noise is removed and the features of the two domains are deeply interacted, and finally reconstructed output through inverse transform.The application blocks the pollution of high-frequency details injection to spectrum from physical logic, significantly improves the spatial definition and spectral fidelity of high-resolution multispectral image.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically, to a method for panchromatic sharpening of remote sensing images based on dual-domain collaborative learning. Background Technology

[0002] With the rapid development of Earth observation technology, remote sensing imagery plays an irreplaceable foundational role in core areas of national economy and national security, such as natural resource surveys, urban planning, disaster emergency monitoring, agricultural and forestry yield estimation, and military reconnaissance. However, due to inherent bottlenecks in the physical hardware of spaceborne optical sensors (such as limitations in light intake and signal-to-noise ratio), a single sensor often cannot simultaneously acquire image data with both high spatial and high spectral resolution. In practical engineering acquisition, systems typically adopt a compromise strategy: separately acquiring panchromatic (PAN) images with high spatial resolution but lacking color spectral information, and multispectral (MS) images with rich multi-band spectral features but lower spatial resolution. In downstream operational applications (such as refined land cover classification and target recognition), simple low-resolution multispectral images cannot provide accurate geometric textures and edge contours, while panchromatic images cannot meet the requirements for physicochemical analysis of surface material composition. Therefore, how to efficiently and faithfully fuse two types of images through panchromatic sharpening to generate high-resolution multispectral (HRMS) images that retain the extreme spatial details of panchromatic images while perfectly inheriting the spectral fidelity of multispectral images, thus breaking the physical limits of satellite sensors, has become a pressing industry challenge in the field of remote sensing image processing. In areas with complex surface coverage (such as high-density urban building clusters or areas with overlapping vegetation), how to completely eliminate "spectral distortion" and "spatial artifacts" while injecting high-frequency details is the biggest bottleneck currently facing the engineering implementation of panchromatic sharpening.

[0003] To address these issues, various panchromatic sharpening techniques have been developed in the industry. Early conventional techniques primarily relied on traditional algorithms such as component substitution (CS) and multi-resolution analysis (MRA). The CS method separates spatial and spectral components through spatial projection and replaces them with panchromatic images. While computationally simple, it inevitably disrupts the original spectral balance, leading to severe spectral distortion. The MRA method relies on filters to extract high-frequency information from panchromatic images and injects it into multispectral images. Although it can preserve spectral characteristics relatively well, it is prone to smoothing and losing spatial details. In recent years, with the evolution of deep learning, panchromatic sharpening models based on convolutional neural networks (CNNs), generative adversarial networks (GANs), and even trans-form architectures have emerged. These conventional deep learning architectures typically perform a simple concatenation of upsampled low-resolution multispectral images and panchromatic images along the channel dimension. Subsequently, stacked convolutional layers or self-attention mechanisms are used to perform nonlinear mapping of local or global features within a "single spatial domain" to implicitly learn the correspondence between images and reconstruct the target image.

[0004] Currently, the industry typically employs a dual-domain learning approach combining the frequency and spatial domains to address these issues. For example, Chinese invention patent CN118587097B discloses a multispectral remote sensing image enhancement method based on frequency-spatial dual-domain learning. It describes performing Fourier transforms on the panchromatic image and the upsampled low-resolution multispectral image to decompose them into phase and amplitude components, learning them separately in the frequency domain, and then reconstructing them using inverse Fourier transform. Simultaneously, it performs channel stitching to extract features from the panchromatic and multispectral images in the spatial domain, and finally performs spatial-frequency fusion. However, this scheme has the following drawbacks: First, in the spatial-frequency dual-domain feature fusion stage, it only uses a relatively static network structure or one lacking clear physical constraints for feature overlay, failing to truly explore the independent value of the phase component (dominantly representing image spatial structure and edge texture) and amplitude component (dominantly representing color distribution and low-frequency energy) in cross-modal fusion. This leads to distortion phenomena where spectral information is contaminated by high-frequency energy when injecting high-frequency spatial details into multispectral images. Second, in the joint representation of panchromatic and multispectral images, a large amount of redundant and conflicting information inevitably exists between the two modes. This scheme lacks an explicit, adaptive threshold-driven cross-domain complementarity rule to accurately quantify and separate the complementary and redundant components between spatial and frequency domain features. This coarse fusion logic makes the network unable to actively remove noise interference in dynamically changing and complex terrain scenes, resulting in serious problems such as frequent structural artifacts, color overflow, and local distortion in spectral gradient regions or areas with dense high-frequency textures, making it difficult to meet the actual needs of high-precision remote sensing interpretation.

[0005] In summary, overcoming the problems of imprecise physical logic of spatial-frequency dual-domain feature interaction (easily leading to spectral distortion) and lack of adaptive cross-domain complementary threshold screening mechanism in existing technologies (easily generating artifacts and redundant interference), and proposing a more efficient and accurate full-color sharpening method and device based on adaptive threshold cross-domain complementary rules and spatial-frequency dual-domain collaboration has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] To address the problems of spectral distortion caused by the imprecise physical logic of spatial-frequency dual-domain feature interaction in existing panchromatic sharpening techniques, and the resulting structural artifacts and redundant interference due to the lack of cross-domain feature filtering mechanisms, this invention proposes a panchromatic sharpening method based on spatial-frequency dual-domain collaboration and adaptive threshold cross-domain complementarity. This method precisely separates panchromatic and multispectral images into Fourier phase and amplitude components with clear physical meaning. An adaptive threshold rule is introduced between the spatial and frequency domains to dynamically quantize, eliminate redundant features, and deeply fuse complementary information. This solves the problem of spectral contamination and local distortion easily caused by injecting high-frequency spatial details in complex terrain scenes, ultimately achieving efficient and accurate reconstruction of remote sensing images with both high spatial resolution and high spectral fidelity. To achieve the above objectives, this invention provides a remote sensing image panchromatic sharpening method based on dual-domain collaborative learning, including the following steps: S1: Acquire the input high-resolution panchromatic image With low-resolution multispectral imagery The low-resolution multispectral image is upsampled to align its spatial dimensions with the panchromatic image, and shallow panchromatic features are extracted separately. and multispectral features ; S2: Extract the shallow panchromatic features With multispectral features The input is fed into the frequency domain branch and, after Fourier transform, decomposed into phase and amplitude components. The phase components of the two modes are concatenated to preserve global structural information. The amplitude components of the two modes are summed and subjected to global average pooling to generate attention weights. These attention weights are then used to weight and fuse the amplitude components of both modes, resulting in a fused amplitude component. The specific calculation formula is as follows: ; in, and These represent the amplitude components of the multispectral and panchromatic features, respectively. and These represent the corresponding attention weight coefficients; The fused amplitude component and the stitched phase component are respectively processed through convolutional layers and activation functions to adjust the channel dimensions, and then reconstructed through inverse Fourier transform to obtain the frequency domain features. ; S3: Extract the shallow panchromatic features With multispectral features The input is fed into the spatial domain branch, where multi-source local features, including local bias, gradient features, and pooling features, are extracted for each modality feature. Among these, the local bias features... The mean square deviation between each pixel value of the input feature for each modality and its local mean is calculated. The specific calculation formula is as follows: ; in, This represents the panchromatic or multispectral feature map input to the spatial domain branch. This represents the local mean feature obtained after the feature map has undergone local average pooling. The extracted multi-source local features from each modality are concatenated to generate spatial attention weights, which are then compared with the corresponding input features. The system performs cross-interactions and then fuses the global context information extracted by the global modeling module to output spatial domain features. ; S4: Frequency domain characteristics of the input step S2 output Spatial domain features output from step S3 After extracting features in the spatial domain and frequency domain through convolutional layers, instance normalization is performed to obtain the weight parameters of each channel feature. The formula for instance normalization is as follows: ; in, This represents the spatial or frequency domain channel characteristics input to the interactive module. This represents the output after instance normalization. The learnable weight parameters represent the features of each channel. This represents the mean. Represents variance. Indicates the offset parameter. Indicates a preset constant; Based on the obtained weight parameters Calculate the adaptive threshold The calculation formula is as follows: ; in, Indicates the weighting factor; The weight parameters of each channel are compared with the adaptive threshold, and a cross-domain feature complementation strategy is executed. The specific calculation formula is as follows: For spatial domain features, the calculation formula is: ; For frequency domain characteristics, the calculation formula is: ; in, and These represent channel features with weights less than the threshold and channel features with weights greater than or equal to the threshold in the spatial domain, respectively. and These represent channel features with weights less than a threshold and channel features with weights greater than or equal to a threshold in the frequency domain, respectively. This represents the frequency domain features of the channel corresponding to the low-weight spatial domain features; This represents the spatial domain features of the channel corresponding to the low-weight frequency domain features; and These represent the adaptive thresholds in the spatial and frequency domains, respectively. Represents element-wise multiplication; The bi-domain features after cross-domain complementation are spliced ​​and fused to output the space-frequency fusion features. S5: Input the spatial-frequency fusion features output from step S4 into the reconstruction module for processing, and add the processing result to the upsampled low-resolution multispectral image from step S1 to output a high-resolution multispectral image. .

[0007] In a preferred embodiment of the present invention, in step S2, the amplitude components of the two modes are summed to obtain the aggregated amplitude feature U; subsequently, a global average pooling operation is used to obtain the spatial global representation, which is then passed sequentially through a first convolutional layer and a second convolutional layer with the LeakyReLU activation function, and finally attention weights are generated using the Softmax function. and The formula for calculating the network operations that generate attention weights is: ; in, This indicates a global average pooling operation. express Convolution operation, This represents the LeakyReLU activation function. This represents the Softmax activation function.

[0008] As a preferred embodiment of the present invention, the process of extracting multi-source local features in step S3 includes: For the shallow panchromatic or multispectral features input to the spatial domain branch, local max pooling, local average pooling, and gradient extraction based on the Sobel operator are performed respectively to obtain max pooling features. Average pooling characteristics With gradient features Combine max pooling features, average pooling features, gradient features, and local bias features. Concatenation is performed along the channel dimension to construct a multi-source feature representation. The calculation formula for its channel splicing is as follows: ; in, This indicates a channel-level join operation.

[0009] As a preferred embodiment of the present invention, the specific process of generating spatial attention weights and interacting with input features in step S3 is as follows: The concatenated multi-source feature representation Passing in sequence Convolutional layers and Convolutional layers are used, and spatial attention weights are generated via a sigmoid activation function. ; Spatial attention weights are combined with shallow panchromatic or multispectral features input to the spatial domain branch. Element-wise multiplication is performed, and residual connections are introduced and added to the input features to obtain local interaction features. The calculation formula is as follows: ; in, These are local interactive features of panchromatic or multispectral images.

[0010] As a preferred embodiment of the present invention, the specific process of fusing the global context information extracted by the global modeling module in step S3 is as follows: using the Mamba module to analyze local interaction features. Global context modeling is performed, and the output global features are added to the local interaction features. Then, the result is compared with the input features from the input to the spatial domain branch in the channel dimension. To assemble; then through Convolution adjusts the channel dimensions to obtain single-modal spatial domain features. The calculation formula is: ; in, This indicates a Mamba module; The single-modal spatial domain features output from the panchromatic image branch and the multispectral image branch are added together to obtain the final spatial domain features. .

[0011] As a preferred embodiment of the present invention, in step S4, the process of splicing and fusing the dual-domain features after cross-domain complementarity is as follows: the complementary and recombined spatial domain features and Channel splicing is performed, and the frequency domain features after complementary recombination are combined. and Perform channel splicing; then reconnect the spliced ​​features through channels, and... The convolutional layer outputs the final spatial-frequency fusion feature, calculated using the following formula: ; in, This is a feature of spatial-frequency fusion.

[0012] As a preferred embodiment of the present invention, the reconstruction module in step S5 is characterized by: The network consists of convolutional layers and the LeakyReLU activation function; during the network model training phase, the following is used: The norm is used as a loss function to constrain model optimization. The formula for calculating the loss function is: ; in, This represents true high-resolution multispectral imagery. This represents the fusion prediction result obtained through the dual-domain collaborative learning network. represent Norm.

[0013] As a preferred embodiment of the present invention, steps S2, S3, and S4 together constitute three cascaded dual-domain collaborative interaction stages, wherein the first... The spatial domain and frequency domain characteristics of the stage output are used as the first... The input features of each stage are respectively input into steps S2 and S3 for further interactive fusion, wherein, After three stages of interaction and fusion, the spatial-frequency fusion features output from the third stage are input into the reconstruction module in step S5.

[0014] As a preferred embodiment of the present invention, in step S1, shallow multispectral features are extracted. With panchromatic features The calculation formula for network operations is as follows: ; in, This represents a low-resolution multispectral image after a 4x upsampling. express Convolution operation, This represents the LeakyReLU activation function; In step S2, the formula for calculating the channel dimension adjustment performed before the inverse Fourier transform is: ; Among them, A MP Indicates the amplitude component, This represents the phase components after channel splicing. This indicates the inverse Fourier transform operation.

[0015] Compared with the relevant prior art, the beneficial effects of the present invention are: The invention innovatively proposes a multi-level dual-domain interactive network (MDI-Net), overcoming the limitation of most existing panchromatic sharpening methods that only perform feature fusion in a single spatial or frequency domain. By designing a dedicated space-frequency interaction module, a cross-domain information exchange mechanism is established between the spatial and frequency domains, achieving complementary advantages and progressive optimization of features.

[0016] In spatial domain processing, this invention cleverly combines the ability of CNNs to extract local high-frequency details with the high efficiency of Mamba models in long-sequence global context modeling. Simultaneously, a spatial attention mechanism is employed to precisely guide effective information interaction between panchromatic and multispectral modalities, avoiding interference from redundant features and noise, and improving the accuracy of spatial structure reconstruction.

[0017] By employing Fourier transform, the image is physically decoupled into amplitude and phase spectra. While strictly preserving the phase spectrum of the dominant global image structure, only the amplitude information originating from the two modes is dynamically weighted and fused. This mechanism fundamentally blocks the contamination path to low-frequency spectral information caused by the injection of high-frequency spatial details, greatly ensuring the spectral fidelity of the multispectral data.

[0018] Experimental data objectively confirm the technical advantages of this invention. Compared with existing comparative methods, this invention achieves a significant improvement in several mainstream evaluation indicators, particularly a 27.54% improvement in the SAM indicator (representing spectral fidelity), a 29.83% improvement in the RMSE indicator (representing reconstruction error), and a 12.32% improvement in the PSNR indicator (representing peak signal-to-noise ratio). The final fusion result not only presents richer texture details visually but also perfectly preserves the spectral characteristics of the multispectral image, achieving extremely high spatial and spectral overall quality. Attached Figure Description

[0019] Figure 1 A schematic diagram of the process for a panchromatic sharpening method for remote sensing images based on dual-domain collaborative learning provided by the present invention; Figure 2 This is an overall fusion framework diagram provided in the embodiments of the present invention; Figure 3 This is a detailed structural diagram of the frequency domain interaction module provided in this embodiment of the invention; Figure 4 This is a structural diagram of the spatial domain interaction module provided in this embodiment of the invention; Figure 5 This is a detailed structural diagram of the space-frequency interaction module provided in this embodiment of the invention; Figure 6 This refers to the specific network structure of the Mamba module provided in this embodiment of the invention. Detailed Implementation

[0020] The solutions provided by the present invention will be further described below with reference to the accompanying drawings. However, the present invention can be implemented in many different ways and should not be construed as limited to the embodiments shown; rather, these embodiments provide those skilled in the art with implementation methods that meet applicable legal requirements.

[0021] Example 1: As Figure 1 As shown, this embodiment of the invention provides a method for panchromatic sharpening of remote sensing images based on spatial-frequency dual-domain collaborative learning. The method provided in this embodiment can be applied to various electronic devices with image data processing capabilities, such as high-performance servers equipped with graphics processing units (GPUs) or tensor processing units (TPUs), cloud computing nodes, or local workstations, etc. This invention does not specifically limit these applications. The core architecture of this embodiment relies on a multi-level dual-domain interactive network (MDI-Net) for pansharpening. This network fully combines the advantages of convolutional neural networks (CNNs), state-space models (Mamba), and Fourier transforms, collaboratively utilizing information from both the spatial and frequency domains.

[0022] Specifically, combined Figure 1 The flowchart shown illustrates the method for panchromatic sharpening of remote sensing images in this embodiment, which specifically includes the following steps: like Figure 2 As shown, this embodiment first preprocesses the input data. Specifically, the LRMS image is upsampled using bicubic interpolation to align its spatial dimensions with the PAN image, while preserving the original PAN image to fully utilize its rich spatial detail. Subsequently, shallow features are extracted from both the upsampled LRMS image and the original PAN image using 3×3 convolutional blocks with the LeakyReLU activation function. and This process can be represented by the following formula: ; in, and These represent the input LRMS and PAN images, respectively. This indicates a 4x upsampling operation. express Convolution operation, This represents the LeakyReLU activation function.

[0023] (1) Frequency domain interaction module: such as Figure 3 As shown, in the frequency domain, this embodiment first analyzes the characteristics of MS and PAN images. and ( Perform a Fourier Transform (FT) on each component to obtain its respective amplitude (A) and phase (P) components. Their respective Fourier Transforms can be expressed by the formula: ; in, and These represent the inputs up to the [number]th [number]. ( LRMS and PAN image features of the phase module, This indicates the Fourier transform operation. Indicates the amplitude component, This represents the phase component.

[0024] In frequency domain fusion, the phase component P carries crucial structural information. Therefore, this embodiment fuses the phase components of both images through channel stitching to ensure that the fusion result fully preserves the structural features of the input image. In contrast, the information contained in the amplitude component varies significantly across different modalities. To more effectively fuse this complementary information, this embodiment designs a selective interaction strategy based on an attention mechanism. Specifically, the two amplitude components are first summed and aggregated; then, a global average pooling is used to obtain a spatial global representation, and a LeakyReLU activation function is utilized. Convolution further enhances the expressive power of frequency domain features; based on this, attention weights are generated through the Softmax function, and the amplitude components of PAN and LRMS images are weighted and fused separately. This process is expressed by the following formula: ; in, This indicates a concatenation operation on the channel dimension of features. This indicates a global average pooling operation. express Convolution operation, This represents the Softmax activation function. and This represents the attention weighting coefficient; and They represent the first ( The fusion results of the phase component and amplitude component at each stage.

[0025] Finally, through the LeakyReLU activation function Convolution uniformly adjusts the channel dimensions of the fused amplitude and phase components, and then uses the inverse Fourier transform (IFT) to reconstruct the final fused frequency domain feature representation into the spatial domain. This process is expressed by the following formula: ; in This indicates the inverse Fourier transform operation. Indicates the first Output of the frequency domain interaction module.

[0026] (2) Spatial domain interaction module: such as Figure 4 As shown, in the spatial domain, to effectively enhance the expressive power of deep features, this embodiment introduces a dual feature extraction mechanism based on local deviation and gradient operators. Specifically, the deviation feature effectively captures subtle intensity changes in the image by calculating the mean square deviation between each pixel value of the input feature and its local mean; simultaneously, the Sobel gradient operator is used to extract spatial texture information, highlighting edge structure features. Subsequently, the obtained deviation feature, gradient feature, and features from Local Max Pooling (LMP) and Local Average Pooling (LAP) are concatenated along the channel dimension to construct a multi-source feature representation. This process is expressed by the following formula: ; in, This represents a local max pooling operation. This indicates a local average pooling operation. This represents gradient computation based on the Sobel operator; Indicates input up to the number Panchromatic or multispectral feature maps of the spatial domain interaction module; , , and They represent the first The feature map obtained in the spatial domain interaction module through local max pooling, local average pooling, Sobel gradient operator operation and bias calculation operation.

[0027] Subsequently, the convolutional layers are used to process the stitched multipath features. Local feature extraction and channel number adjustment are performed, and spatial attention weights are generated via a sigmoid activation function. These attention weights are then compared with the input features. Cross-domain interactions are performed. Furthermore, residual connections are introduced during spatial domain interactions to alleviate the vanishing gradient problem. This process is expressed by the following formula: ; in, This represents the Sigmoid activation function. This represents the spatial attention weight coefficient; This represents element-wise multiplication.

[0028] To further enhance global modeling capabilities, such as Figure 6 As shown, this embodiment introduces the Mamba module to capture global context information and interacts with and fuses it with local features. Finally, the features of the two branches are adjusted for channel dimensions through 1×1 convolutions and then fused together to obtain the final output of the spatial domain interaction module. This process is expressed by the following formula: ; in, This represents a Mamba block for extracting global features.

[0029] (3) Spatial-Frequency Interaction Module: In the spatial-frequency interaction module, to achieve effective fusion of the two modal features in the spatial and frequency domains, this embodiment introduces an adaptive threshold strategy based on weighting factors. This strategy can automatically evaluate the importance of each channel feature, thereby fully integrating the complementary information of the two domains. The calculation process of the adaptive threshold based on weighting factors can be expressed by the formula: ; in, In this invention, the weighting factor is represented. . This represents the weight of each channel of the input features, and its magnitude reflects the importance of that channel's feature in contributing to the final fusion result. Weight The channel information is derived from the input features learned through the instance normalization layer, where the instance normalization operation is as follows: ; in, This represents the feature maps of each channel in the spatial or frequency domain input to the space-frequency interaction module. This represents the output after instance normalization. Learnable weight parameters representing the characteristics of each channel in the spatial or frequency domain. The mean value representing the characteristics of each channel in the spatial or frequency domain. The variance representing the characteristics of each channel in the spatial or frequency domain. The learnable offset parameter represents the characteristics of each channel in the spatial or frequency domain. This represents a constant with a very small value.

[0030] like Figure 5 As shown, in the space-frequency interaction module, this embodiment will use features from the dual domains. and Feature extraction is performed through convolutional layers, followed by processing through instance normalization layers to obtain the weight parameters of each channel feature. To achieve complementary enhancement of dual-domain information, this embodiment designs a cross-domain feature interaction strategy. Specifically, for a low-weight channel feature (weight less than a threshold) in one domain, complementary enhancement is achieved by introducing feature information from the corresponding channel in another domain, thereby improving the expressive power of the low-weight feature. This process is expressed by the following formula: ; in, and These represent channels in the spatial domain. Low weight and channel High-weight features, and These represent the channels in the frequency domain. Low weight and channel High-weight features; This represents the frequency domain characteristics of the channel corresponding to the low-weight spatial domain characteristics. This represents the spatial domain features of the channel corresponding to the low-weight frequency domain features; This indicates the instance normalization operation. and These represent the adaptive thresholds in the spatial and frequency domains, respectively.

[0031] Finally, the features obtained through the three-stage interaction and fusion are fed into a reconstruction module consisting of 1×1 convolutions and the LeakyReLU activation function for processing, and supplemented with the upsampled LRMS image to obtain the final fusion result. This process can be represented by the following formula: ; in, This indicates the result after processing by the reconstruction module. This indicates the result of full-color sharpening.

[0032] Example 2: To further verify the effectiveness and advancement of the remote sensing image panchromatic sharpening method based on spatial-frequency dual-domain collaborative learning (i.e., multi-level dual-domain interactive network MDI-Net) proposed in Example 1 of this invention, this example provides a specific experimental verification and comparative analysis process. It should be understood that the experimental parameters and datasets in this example are merely illustrative and do not constitute a limitation on the scope of protection of this invention.

[0033] This embodiment's experiments are based on publicly available remote sensing image datasets (such as mainstream satellite datasets like GaoFen-2, WorldView-3, or QuickBird). The experiments were conducted on a deep learning workstation equipped with a high-performance graphics processing unit (GPU). The acquired public datasets were divided into training, validation, and test sets according to a proportional division principle. During training, Wald's Protocol, simulating real-world scenarios, was used to spatially downsample the original high-resolution panchromatic (PAN) and multispectral (MS) images to construct training sample pairs with ground truth values. In the testing phase, the original resolution images were directly used for full-resolution experimental validation.

[0034] To comprehensively evaluate the performance of the proposed method, representative panchromatic sharpening methods in the field were selected as baseline comparison methods, including traditional component substitution methods (such as Brovey), multi-resolution analysis methods, and mainstream deep learning-based methods (such as CNN-based MSDCNN and Transformer-based LFormer). Regarding objective evaluation metrics, this embodiment selected seven core metrics to comprehensively measure image spatial detail and spectral fidelity: metrics reflecting spatial structure preservation ability: Multiscale Structural Similarity Index (MS-SSIM, higher values ​​are better), Spatial Correlation Coefficient (SCC, higher values ​​are better); metrics reflecting overall image reconstruction quality: Peak Signal-to-Noise Ratio (PSNR, higher values ​​are better), Universal Image Quality Index (UIQI, higher values ​​are better); metrics reflecting spectral fidelity and overall error: Spectral Angle Mapper (SAM, lower values ​​are better, assessing spectral distortion), Root Mean Square Error (RMSE, lower values ​​are better), and Relative Dimensionless Global Error (ERGAS, lower values ​​are better).

[0035] Based on the above experimental setup, the method proposed in this invention and the aforementioned comparative method were tested on the same test set using full-color sharpening reconstruction. The experimental results demonstrate the significant superiority of the method proposed in this invention from both subjective visual effect and objective quantitative evaluation dimensions. Subjective visual effect comparison: By visually interpreting and comparing the generated HRMS images with local magnification, the results show that, compared with existing comparison methods, the method proposed in this invention benefits from the accurate preservation of the phase spectrum (dominant global structure) in the frequency domain and the deep fusion of CNN and Mamba features in the spatial domain. The resulting fusion result has richer texture information, sharper ground object edges, and eliminates common spatial blur and jagged artifacts. At the same time, thanks to the adaptive weighting of amplitude information in the cross-domain complementarity rule, the spectral pollution during high-frequency detail injection is greatly reduced, and the image colors are more natural and realistic. The subjective visual effect is significantly better than other comparison methods.

[0036] Comparison of objective evaluation indicators (quantitative analysis): Objective evaluation indicator data on the test set were statistically analyzed to calculate the average performance improvement of the method of this invention compared to all comparable methods. Quantitative results show that: In terms of spatial structure reconstruction: the multi-scale structural similarity index (MS-SSIM) of the method of the present invention is about 6.14% better than the average of the comparative methods, and the spatial correlation coefficient (SCC) is about 5.37% better than the average of the comparative methods, which proves the efficiency of the present invention in extracting and injecting spatial details of panchromatic images; In terms of overall image quality and signal-to-noise ratio: the peak signal-to-noise ratio (PSNR) of the method of the present invention is about 12.32% better than the average of the comparison methods, and the universal image quality index (UIQI) is about 4.67% better than the average of the comparison methods, indicating that the generated image has a higher signal-to-noise ratio and extremely low redundant noise; In terms of spectral fidelity and error control (core advantages): the spectral angle mapper (SAM) error of the method of this invention is about 27.54% better than the average of the comparative methods (significantly reduced error), the root mean square error (RMSE) is about 29.83% better than the average of the comparative methods, and the relative dimensionless global error (ERGAS) is about 24.48% better than the average of the comparative methods. These three indicators involving global error and spectral distortion show a cliff-like optimization of nearly 30%, which fully verifies the decisive role of the "spatial-frequency dual-domain synergy and phase / amplitude decoupling" mechanism in overcoming the pain point of "spectral distortion" in traditional methods.

[0037] In summary, the experimental comparison in this embodiment shows that the fusion result obtained by the method proposed in this invention not only effectively preserves the extreme spatial information of the panchromatic image, but also maintains the spectral characteristics of the multispectral image to a very high degree. It perfectly solves the spectral distortion problem caused by insufficient interaction of spatial and frequency features, and has extremely significant progress and broad engineering application prospects.

[0038] The above embodiments merely illustrate implementation methods of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A panchromatic sharpening method for remote sensing images based on dual-domain collaborative learning, characterized in that: The method includes the following steps: S1: Acquire the input high-resolution panchromatic image With low-resolution multispectral imagery The low-resolution multispectral image is upsampled to align its spatial dimensions with the panchromatic image, and shallow panchromatic features are extracted separately. and multispectral features ; S2: Extract the shallow panchromatic features With multispectral features The input is fed into the frequency domain branch and decomposed into phase and amplitude components by Fourier transform. The phase components of the two modal features are then concatenated to preserve global structural information. The amplitude components of the two modes are summed and subjected to global average pooling to generate attention weights. The amplitude components of the two modes are then weighted and fused using these attention weights to generate the fused amplitude components. The specific calculation formula is as follows: ； in, and These represent the amplitude components of the multispectral and panchromatic features, respectively. and These represent the corresponding attention weight coefficients; The fused amplitude component and the stitched phase component are respectively processed through convolutional layers and activation functions to adjust the channel dimensions, and then reconstructed through inverse Fourier transform to obtain the frequency domain features. ; S3: Extract the shallow panchromatic features With multispectral features The input is fed into the spatial domain branch, where multi-source local features, including local bias, gradient features, and pooling features, are extracted for each modality feature. Among these, the local bias features... The mean square deviation between each pixel value of the input feature for each modality and its local mean is calculated. The specific calculation formula is as follows: ； in, This represents the panchromatic or multispectral feature map input to the spatial domain branch. This represents the local mean feature obtained after the feature map has undergone local average pooling. The extracted multi-source local features from each modality are concatenated to generate spatial attention weights, which are then compared with the corresponding input features. The system performs cross-interactions and then fuses the global context information extracted by the global modeling module to output spatial domain features. ; S4: Frequency domain characteristics of the input step S2 output Spatial domain features output from step S3 After extracting features in the spatial domain and frequency domain through convolutional layers, instance normalization is performed to obtain the weight parameters of each channel feature. The formula for instance normalization is as follows: ； in, This represents the spatial or frequency domain channel characteristics input to the interactive module. This represents the output after instance normalization. The learnable weight parameters represent the features of each channel. This represents the mean. Represents variance. Indicates the offset parameter. Indicates a preset constant; Based on the obtained weight parameters Calculate the adaptive threshold The calculation formula is as follows: ； in, Indicates the weighting factor; The weight parameters of each channel are compared with the adaptive threshold, and a cross-domain feature complementation strategy is executed. The specific calculation formula is as follows: For spatial domain features, the calculation formula is: ； For frequency domain characteristics, the calculation formula is: ； in, and These represent channel features with weights less than the threshold and channel features with weights greater than or equal to the threshold in the spatial domain, respectively. and These represent channel features with weights less than a threshold and channel features with weights greater than or equal to a threshold in the frequency domain, respectively. This represents the frequency domain features of the channel corresponding to the low-weight spatial domain features; This represents the spatial domain features of the channel corresponding to the low-weight frequency domain features; and These represent the adaptive thresholds in the spatial and frequency domains, respectively. Represents element-wise multiplication; The bi-domain features after cross-domain complementation are spliced ​​and fused to output the space-frequency fusion features. S5: Input the spatial-frequency fusion features output from step S4 into the reconstruction module for processing, and add the processing result to the upsampled low-resolution multispectral image from step S1 to output a high-resolution multispectral image. .

2. The remote sensing image panchromatic sharpening method based on dual-domain collaborative learning as described in claim 1, characterized in that: In step S2, the amplitude components of the two modes are summed to obtain the aggregated amplitude feature U; then, a global average pooling operation is used to obtain the spatial global representation, which is then passed through a first convolutional layer and a second convolutional layer with LeakyReLU activation function in sequence, and finally attention weights are generated by the Softmax function. and The formula for calculating the network operations that generate attention weights is: ； in, This indicates a global average pooling operation. express Convolution operation, This represents the LeakyReLU activation function. This represents the Softmax activation function.

3. The remote sensing image panchromatic sharpening method based on dual-domain collaborative learning as described in claim 1, characterized in that: In step S3, the process of extracting multi-source local features includes: For the shallow panchromatic or multispectral features input to the spatial domain branch, local max pooling, local average pooling, and gradient extraction based on the Sobel operator are performed respectively to obtain max pooling features. Average pooling characteristics With gradient features Combine max pooling features, average pooling features, gradient features, and local bias features. Concatenation is performed along the channel dimension to construct a multi-source feature representation. The calculation formula for its channel splicing is as follows: ； in, This indicates a channel-level join operation.

4. The remote sensing image panchromatic sharpening method based on dual-domain collaborative learning as described in claim 1, characterized in that: In step S3, the specific process of generating spatial attention weights and cross-interacting with the input features is as follows: The spliced ​​multi-source feature representation Passing in sequence Convolutional layers and Convolutional layers are used, and spatial attention weights are generated via a sigmoid activation function. ; Spatial attention weights are combined with shallow panchromatic or multispectral features input to the spatial domain branch. Element-wise multiplication is performed, and residual connections are introduced and added to the input features to obtain local interaction features. The calculation formula is as follows: ； in, These are local interactive features of panchromatic or multispectral images.

5. The remote sensing image panchromatic sharpening method based on dual-domain collaborative learning as described in claim 1, characterized in that: In step S3, the specific process of fusing the global context information extracted by the global modeling module is as follows: using the Mamba module to analyze local interaction features. Global context modeling is performed, and the output global features are added to the local interaction features. Then, the result is compared with the input features from the input to the spatial domain branch in the channel dimension. To assemble; then through Convolution adjusts the channel dimensions to obtain single-modal spatial domain features. The calculation formula is: ； in, This indicates a Mamba module; The single-modal spatial domain features output from the panchromatic image branch and the multispectral image branch are added together to obtain the final spatial domain features. .

6. The remote sensing image panchromatic sharpening method based on dual-domain collaborative learning as described in claim 1, characterized in that: In step S4, the process of splicing and fusing the bi-domain features after cross-domain complementation is as follows: the complementary and recombined spatial domain features are... and Channel splicing is performed, and the frequency domain features after complementary recombination are combined. and Perform channel splicing; then reconnect the spliced ​​features through channels, and... The convolutional layer outputs the final spatial-frequency fusion feature, calculated using the following formula: ； in, This is a feature of spatial-frequency fusion.

7. The remote sensing image panchromatic sharpening method based on dual-domain collaborative learning as described in claim 1, characterized in that: The reconstruction module in step S5 is composed of The network consists of convolutional layers and the LeakyReLU activation function; during the network model training phase, the following is used: The norm is used as a loss function to constrain model optimization. The formula for calculating the loss function is: ； in, This represents true high-resolution multispectral imagery. This represents the fusion prediction result obtained through the dual-domain collaborative learning network. represent Norm.

8. The remote sensing image panchromatic sharpening method based on dual-domain collaborative learning as described in claim 1, characterized in that: Steps S2, S3, and S4 together constitute three cascaded dual-domain collaborative interaction stages, among which the first... The spatial domain and frequency domain characteristics of the stage output are used as the first... The input features of each stage are respectively input into steps S2 and S3 for further interactive fusion, wherein, After three stages of interaction and fusion, the spatial-frequency fusion features output from the third stage are input into the reconstruction module in step S5.

9. The remote sensing image panchromatic sharpening method based on dual-domain collaborative learning as described in claim 1, characterized in that: In step S1, shallow multispectral features are extracted. With panchromatic features The calculation formula for network operations is as follows: ； in, This represents a low-resolution multispectral image after a 4x upsampling. express Convolution operation, This represents the LeakyReLU activation function; In step S2, the formula for calculating the channel dimension adjustment performed before the inverse Fourier transform is: ； Among them, A MP Indicates the amplitude component, This represents the phase components after channel splicing. This indicates the inverse Fourier transform operation.

Images