Image fusion method and device based on dual-domain flexible converter for full-color sharpening

Abstract

The application discloses a kind of based on dual-domain flexible converter's panchromatic sharpening image fusion method and its device.The method comprises: to low-resolution multispectral image and high-resolution panchromatic image are carried out multi-layer space-frequency joint attention operation, generate characteristic tensor;Carry out dual-domain feature alignment operation, and based on attention mechanism alignment structure semantic feature, through Fourier transform fusion amplitude and phase spectrum information, utilize instance normalization matching modal distribution difference;Carry out multi-level fusion, each level executes splicing, convolution and space-frequency joint attention operation in turn;Residual reconstruction is carried out to fusion feature;Reconstruction feature is added to low-resolution multispectral image after upsampling element by element, and output high-resolution multispectral image.The application can significantly reduce image structure distortion and detail blur, improve image definition and edge restoration ability, improve fusion consistency and physical authenticity, enhance the detail expressiveness of fusion image.

Description

Technical Field

[0001] This invention relates to a pancolor sharpening image fusion method in the field of image processing technology, and more particularly to a pancolor sharpening image fusion method based on a dual-domain flexible converter, and also to a pancolor sharpening image fusion device based on a dual-domain flexible converter. Background Technology

[0002] Panchromatic sharpening is a key task in remote sensing image processing, aiming to fuse low-resolution multispectral (LRMS) images with high-resolution panchromatic (PAN) images to generate high-resolution multispectral (HRMS) images. This technology is of great significance in remote sensing applications such as land use classification, urban planning, and environmental monitoring. Existing panchromatic sharpening methods can be mainly divided into two categories: traditional model-driven methods and data-driven methods. Traditional methods include the IHS (Intensity-Hue-Saturation) transform [Carper et al., 1990], the Brovy transform [Gillespie, 1987], and principal component analysis (PCA) [Aiazzi et al., 2007]. These methods are computationally efficient, but the fusion results are prone to problems such as spectral distortion and spatial artifacts.

[0003] Convolutional Neural Networks (CNNs) have limited receptive fields, making it difficult to capture long-range dependencies in images, resulting in insufficient structural restoration of generated images. Transformer-based models utilize self-attention mechanisms to model long-range dependencies, improving spatial detail restoration capabilities. However, current mainstream Transformer methods still have the following significant problems: (1) Rigid fixed attention window: Existing methods mostly use fixed-size sliding windows for attention calculation, which is difficult to adapt to the complex and varied landforms in remote sensing images, easily causing loss of local structural information and edge blurring; (2) Failure to jointly model spatial and frequency information: Most methods only model features in the spatial domain, ignoring the importance of the frequency domain (such as texture and high-frequency details), or treating frequency processing as an independent module rather than a whole fusion, resulting in insufficient utilization of spectral information and loss of detail in the fusion results; (3) Insufficient feature alignment between modes: Since PAN and MS images originate from different sensors and have significant modal differences, if there is a lack of effective alignment mechanisms, direct fusion is prone to artifacts or color deviations, affecting image quality. Simultaneously, methods that model spatial and frequency information can help improve image fusion performance, but these methods are mainly aimed at hyperspectral images and lack in-depth research on feature alignment and window adaptation mechanisms in PAN-MS image fusion.

[0004] Therefore, existing panchromatic sharpening methods have problems such as difficulty in adapting to complex ground structures, large feature differences between multi-source remote sensing images, and lack of effective alignment mechanisms. Summary of the Invention

[0005] To address the technical problems of existing panchromatic sharpening methods, such as difficulty in adapting to complex ground structures, large feature differences between multi-source remote sensing images, and lack of effective alignment mechanisms, this invention provides a panchromatic sharpening image fusion method and apparatus based on a dual-domain flexible converter.

[0006] This invention is achieved using the following technical solution: a full-color sharpening image fusion method based on a dual-domain flexible converter, comprising the following steps:

[0007] S1. Feature extraction: Perform multi-layer spatial-frequency joint attention operations on low-resolution multispectral images and high-resolution panchromatic images respectively to generate feature tensors MS and PAN;

[0008] S2. Feature Alignment: Perform dual-domain feature alignment on MS and PAN, and align structural semantic features based on an attention mechanism. Fourier transform is used to fuse amplitude and phase spectrum information, and instance normalization is used to match modal distribution differences. The dual-domain feature alignment is achieved through a spatial attention path and a frequency alignment path, combined with a modulation compensation path, and the outputs of the three paths are fused into aligned features. Specifically, the spatial path generates a query vector from the PAN image and key-value pairs from the MS image; the frequency path performs Fourier transform on PAN and MS, adding amplitude and phase information to reconstruct frequency domain features; the modulation path generates modulation parameters based on the statistical characteristics of MS and normalizes and recalibrates the PAN features.

[0009] S3, Feature Fusion: The aligned features and MS are fused in multiple levels. At each level, concatenation, convolution and spatial-frequency joint attention operations are performed sequentially to obtain the fused features.

[0010] S4. High-frequency reconstruction: Perform residual reconstruction on the fused features to generate a feature map with enhanced high-frequency details;

[0011] S5. Image Generation: Add the reconstructed features to the upsampled low-resolution multispectral image element by element to output a high-resolution multispectral image.

[0012] This invention utilizes a spatial-frequency flexible attention mechanism to dynamically adjust the size and offset of the attention window based on image content, thereby better capturing feature information of irregular regions. By constructing a "dual-domain feature alignment module," contextual attention is introduced in the spatial domain, and Fourier transform paths are introduced in the frequency domain, significantly enhancing the consistency and fusionability of the two types of image features. This solves the technical problems of existing pancolor sharpening methods, such as difficulty in adapting to complex ground structures, large feature differences between multi-source remote sensing images, and lack of effective alignment mechanisms.

[0013] As a further improvement to the above scheme, the space-frequency joint attention operation is implemented in the following way:

[0014] (1) Predict the sampling scale of each window based on input features S i and offset O i Generate adaptive sampling coordinates;

[0015] (2) Sample and project from the key / value tensor, and combine with the query vector. Q Calculate attention output;

[0016] (3) Combine the output results with the high-frequency features extracted by wavelet transform. F frequency Weighted fusion.

[0017] Furthermore, the formula for calculating the adaptive sampling coordinates is as follows:

[0018]

[0019] In the formula, p sample This represents the adaptive sampling coordinates. p base Represents the reference grid coordinates. p ref This represents the reference vector.

[0020] Furthermore, the formula for calculating the spatial alignment features is as follows:

[0021]

[0022] In the formula, F SIM Features indicating spatial alignment K Represents the key matrix, V Represents a value matrix, d k Indicates the dimension of the key / query vector.

[0023] Furthermore, the formula for calculating the fusion feature is as follows:

[0024]

[0025] In the formula, F SFA Indicates fusion characteristics, Kd Indicates to K linear projection, Vd Indicates to V linear projection, B λ represents the position bias term, and λ represents the learnable fusion weights.

[0026] Furthermore, in step S2, the Fourier transform fusion method includes the following steps:

[0027] (1) Perform Fourier transform on MS and PAN to obtain the corresponding complex spectrum:

[0028]

[0029] In the formula, F () represents the Fast Fourier Transform. A () represents the amplitude spectrum. θ () represents the phase spectrum. j Represents the imaginary unit;

[0030] (2) The amplitude spectrum and phase spectrum of the complex spectrum are weighted and fused or directly added to obtain the comprehensive frequency domain:

[0031]

[0032] In the formula, A ( fused ) indicates amplitude spectrum fusion. θ ( fused ) indicates phase spectrum fusion;

[0033] (3) First, recombine the fused amplitude and phase into a complex spectrum, and then obtain the frequency-domain aligned features through inverse Fourier transform:

[0034]

[0035] In the formula, F FIM This represents the features after frequency domain alignment. This indicates the operation of taking the real part. F -1 () indicates the inverse Fourier transform.

[0036] Furthermore, distribution calibration is performed through the modulation mechanism, and the mean and variance of the MS are obtained through channel statistics. Modulation parameters are predicted by splicing the MS and PAN, and the PAN is then reconstructed after instance normalization. The formula for instance-normalized PAN is:

[0037]

[0038] In the formula, F Mod Indicates the characteristics after distribution calibration. σ 2 Represents variance. μ This represents the mean. γ , β Indicates modulation parameters, IN () indicates instance normalization.

[0039] Furthermore, the alignment feature is represented as:

[0040]

[0041] In the formula, F AB This refers to the alignment feature.

[0042] Furthermore, in step S5, a stacked structure is first used for hierarchical fusion to... F ABThe image is input into the fusion network along with the MS image, and the predicted image is output through the residual module. This predicted image is then added to the MS image after bicubic interpolation to output the high-resolution multispectral image. The calculation formula for the high-resolution multispectral image is as follows:

[0043]

[0044] In the formula, R () indicates residual module operation. ГK () represents the fusion network. MS bic This represents a low-resolution multispectral image after upsampling.

[0045] This invention also provides a pan-color sharpening image fusion device based on a dual-domain flexible converter, which applies any of the pan-color sharpening image fusion methods based on dual-domain flexible converters described above; the device includes:

[0046] The feature extraction module is used to perform multi-layer spatial-frequency joint attention operations on low-resolution multispectral images and high-resolution panchromatic images respectively to generate feature tensors MS and PAN.

[0047] The feature alignment module is used to perform dual-domain feature alignment operations on MS and PAN, and align structural semantic features based on the attention mechanism. It fuses amplitude and phase spectrum information through Fourier transform and uses instance normalization to match modal distribution differences.

[0048] The feature fusion module is used to fuse the alignment features and MS in multiple levels. Each level sequentially performs concatenation, convolution and spatial-frequency joint attention operations to obtain the fused features.

[0049] A high-frequency reconstruction module is used to perform residual reconstruction on the fused features to generate a feature map with enhanced high-frequency details.

[0050] The image generation module is used to add the reconstructed features to the upsampled low-resolution multispectral image element by element to output a high-resolution multispectral image.

[0051] Compared with existing pancolor sharpening methods, the pancolor sharpening image fusion method and apparatus based on dual-domain flexible converter of the present invention have the following advantages:

[0052] 1. This pancolor sharpening image fusion method based on a dual-domain flexible transformer solves the problem of rigidly fixed attention windows in existing Transformer methods, which are difficult to adapt to complex terrain structures. Existing Transformer-based pancolor sharpening methods mostly use fixed window sizes for self-attention calculations, lacking adaptability to image content, resulting in insufficient detail modeling capabilities for complex scenes such as terrain contours and building boundaries. This pancolor sharpening image fusion method, through a spatial-frequency flexible attention mechanism, breaks the spatial fixed window limitation of traditional self-attention mechanisms, achieving adaptive perception and modeling of terrain structures. It can dynamically adjust the size and offset of the attention window according to the image content, effectively adapting to complex situations with large scale variations and irregular shapes in remote sensing images, thus better capturing feature information of irregular areas. Compared with fixed-window Transformers, it can significantly reduce image structure distortion and detail blurring.

[0053] 2. This pancolor sharpening image fusion method based on a dual-domain flexible transformer solves the problems of decoupling and insufficient information utilization in spatial and frequency domain modeling in existing methods. Most traditional methods only model features in the spatial domain or treat frequency information as an auxiliary module, failing to achieve joint spatial-frequency optimization. This pancolor sharpening image fusion method introduces discrete wavelet transform and Fourier transform paths in both the feature extraction and fusion stages, effectively fusing high-frequency details and global structural information, improving the texture fidelity and edge sharpness of the fused image. Through inverse transform reconstruction, it effectively compensates for the inherent low-pass characteristic of the Transformer attention mechanism, preserving high-frequency details such as texture and edges in the image, thus improving image sharpness and edge restoration capabilities.

[0054] 3. This panchromatic sharpening image fusion method based on a dual-domain flexible converter addresses the problem of large feature differences and lack of effective alignment mechanisms between multi-source remote sensing images. PAN and MS images, originating from different sensors, exhibit significant modal differences, making direct fusion prone to introducing artifacts and color deviations. Existing methods often only consider spatial domain alignment, neglecting frequency domain feature differences. This panchromatic sharpening image fusion method constructs a "dual-domain feature alignment module," introducing contextual attention in the spatial domain and an FFT (Fast Fourier Transform) path in the frequency domain, significantly enhancing the consistency and fusionability of the two image features. Furthermore, it achieves explicit alignment of the amplitude and phase spectra through Fast Fourier Transform (FFT), combined with an instance normalization modulation strategy, to compensate for distribution differences between modes, thereby improving fusion consistency and physical realism.

[0055] 4. This pancolor sharpening image fusion method based on a dual-domain flexible converter solves the problems of high-frequency detail loss and edge blurring in the fused image. Since the self-attention mechanism is essentially a low-pass filter, it easily erases high-frequency texture information during image processing, leading to a blurred fusion result. This pancolor sharpening image fusion method introduces a wavelet decomposition path to specifically preserve high-frequency components and injects attention output in a learnable manner, effectively enhancing the detail representation of the fused image.

[0056] 5. This panchromatic image fusion method based on a dual-domain flexible transformer addresses the problem of insufficient generalization ability of existing panchromatic sharpening methods across different scenes. Existing methods often perform well on specific datasets, but their adaptability to diverse scenes such as complex terrain, urban buildings, and natural landscapes is poor. This panchromatic image fusion method employs a multi-scale modeling strategy, combined with hierarchical feature extraction and fusion design, demonstrating good robustness and generalization on multiple real remote sensing datasets (such as WorldView-II, GaoFen-2, and WorldView-III). Furthermore, this method achieves modular structure, flexible fusion methods, and high computational efficiency, and is compatible with existing CNN and Transformer architectures. It demonstrates superior subjective and objective image quality metrics compared to existing technologies on multiple publicly available remote sensing image fusion datasets, including PSNR, SSIM, SAM, ERGAS, and QNR, with significant advantages, particularly in spectral fidelity and spatial clarity. It possesses strong engineering practical value and promotion potential.

[0057] 6. The pancolor sharpening image fusion device based on the dual-domain flexible converter has the same beneficial effects as the pancolor sharpening image fusion method described above, and will not be repeated here. Attached Figure Description

[0058] Figure 1 This is a flowchart of the full-color sharpening image fusion method based on a dual-domain flexible converter according to Embodiment 1 of the present invention.

[0059] Figure 2 This is an internal structure diagram of a spatial-frequency attention module used in the full-color sharpening image fusion method of Embodiment 1 of the present invention.

[0060] Figure 3 This is a schematic diagram of the overall network architecture DDFformer used in the full-color sharpening image fusion method of Embodiment 1 of the present invention.

[0061] Figure 4 This is a schematic diagram of the dual-domain feature alignment module (FAB) used in the full-color sharpening image fusion method of Embodiment 1 of the present invention.

[0062] Figure 5This is a comparison image of the full-color sharpening image fusion method of Embodiment 1 of the present invention and fused images generated by different fusion methods.

[0063] Figure 6 This is a data comparison chart showing the panchromatic sharpening image fusion method of Embodiment 1 of the present invention and the objective evaluation indicators obtained by different methods on the WorldView-III remote sensing image dataset.

[0064] Figure 7 This is a residual heatmap between images generated by different methods and a reference image on the WorldView-III dataset in Embodiment 1 of the present invention.

[0065] Figure 8 This is an experimental result of ablation of the two key modules, SFFA and FAB, in the full-color sharpening image fusion method of Embodiment 1 of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0067] Example 1

[0068] Please see Figure 1 This embodiment provides a panchromatic image fusion method based on a dual-domain flexible converter. The purpose of this method is to fuse a low-resolution multispectral image (LRMS) with a high-resolution panchromatic image (PAN) to generate a high-resolution multispectral image (HRMS). In this embodiment, the panchromatic image fusion method includes the following steps (S1-S5).

[0069] S1. Feature Extraction: Multi-layer spatial-frequency joint attention operations are performed on the low-resolution multispectral image and the high-resolution panchromatic image respectively to generate feature tensors MS and PAN. In this embodiment, the introduced spatial-frequency flexible attention module (SFFA) achieves adaptive spatial perception through a flexible window mechanism.

[0070] S2. Feature Alignment: A dual-domain feature alignment operation is performed on MS and PAN, and structural semantic features are aligned based on an attention mechanism. Amplitude and phase spectrum information are fused using Fourier transform, and instance normalization is used to match modal distribution differences. This embodiment uses a dual-path approach to perform spatial and frequency-based alignment of MS and PAN modal features. In the spatial path, input features achieve context completion through an attention mechanism. The process is as follows: a query Q = Conv(PAN) is constructed, where both the key and value are extracted from MS features.

[0071] In this embodiment, the feature calculation formula for spatial alignment is:

[0072]

[0073] In the formula, F SIM Features indicating spatial alignment K Represents the key matrix, V Represents a value matrix, d k Indicates the dimension of the key / query vector.

[0074] In this embodiment, in the frequency path, the system uses Fast Fourier Transform (FFT) to extract frequency features from the high-resolution panchromatic image (PAN) and the low-resolution multispectral image (MS). Specifically, the Fourier transform fusion method includes the following steps:

[0075] (1) Perform Fourier transform on MS and PAN to obtain the corresponding complex spectrum:

[0076]

[0077] In the formula, F () represents the Fast Fourier Transform, and A() represents the amplitude spectrum. θ () represents the phase spectrum. j Represents the imaginary unit;

[0078] (2) The amplitude spectrum and phase spectrum of the complex spectrum are weighted and fused or directly added to obtain the comprehensive frequency domain:

[0079]

[0080] In the formula, A ( fused ) indicates amplitude spectrum fusion. θ ( fused ) indicates phase spectrum fusion;

[0081] (3) First, recombine the fused amplitude and phase into a complex spectrum, and then obtain the frequency-domain aligned features through inverse Fourier transform:

[0082]

[0083] In the formula, F FIM This represents the features after frequency domain alignment. This indicates the operation of taking the real part. F -1() indicates the inverse Fourier transform. This frequency alignment path can effectively fuse frequency component information from MS and PAN, especially showing better alignment in high-frequency regions such as edges and textures, which helps to enhance the clarity and structural consistency of the final fused image.

[0084] S3. Feature Fusion: The aligned features and MS are fused in multiple stages. Each stage sequentially performs concatenation, convolution, and spatial-frequency joint attention operations to obtain fused features.

[0085] Traditional Transformer fixed windows have difficulty capturing irregular objects. In this embodiment, spatial-frequency joint attention operation is implemented through the lightweight network Fpred, mainly through the following methods: (1) Predict the sampling scale of each window based on input features. S i and offset O i (2) Sample and project from the key / value tensor, and combine with the query vector to generate adaptive sampling coordinates; Q Calculate the attention output; (3) Combine the output with the high-frequency features extracted by wavelet transform (e.g., Haar wavelet or other wavelets). F frequency Weighted fusion.

[0086] The formula for calculating adaptive sampling coordinates is:

[0087]

[0088] In the formula, p sample Indicates adaptive sampling coordinates. p base Represents the reference grid coordinates. p ref This represents the reference vector.

[0089] The final fusion feature is obtained, calculated using the following formula:

[0090]

[0091] In the formula, F SFA Indicates fusion characteristics, Kd Indicates to K linear projection, Vd Indicates to V linear projection, B λ represents the position bias term, and λ represents the learnable fusion weights.

[0092] S4. High-Frequency Reconstruction: Residual reconstruction is performed on the fused features to generate a feature map with enhanced high-frequency details. In this embodiment, to further reduce modal differences, distribution calibration is performed through a modulation mechanism, and the mean and variance of the MS are obtained through channel statistics. Modulation parameters are predicted by splicing the MS and PAN, and then the PAN is reconstructed after instance normalization. The formula for instance normalization of the PAN is:

[0093]

[0094] In the formula, F Mod Indicates the characteristics after distribution calibration. σ 2 Represents variance. μ This represents the mean. γ , β Indicates modulation parameters, IN () indicates instance normalization.

[0095] Therefore, the alignment feature of the final output is:

[0096]

[0097] In the formula, F AB This indicates alignment features.

[0098] S5. Image Generation: The reconstructed features are added element-wise to the upsampled low-resolution multispectral image to output a high-resolution multispectral image. In this embodiment, a stacked structure is first used for hierarchical fusion to... F AB The MS image is input into the fusion network along with the image source, and the predicted image is output through the residual module. This predicted image is then added to the MS image after bicubic interpolation to output a high-resolution multispectral image. This fusion strategy enables the system to have structural adaptability in both the spatial and frequency domains, accurately reproducing ground object edges and spectral details, and demonstrating excellent subjective and objective evaluation performance in field measurements. The formula for calculating the high-resolution multispectral image is as follows:

[0099]

[0100] In the formula, R () indicates residual module operation. ГK () represents a converged network. MS bic This represents a low-resolution multispectral image after upsampling.

[0101] Please see Figure 2The figure illustrates the internal structure of a spatial-frequency attention module, which implements joint spatial-frequency attention operations. First, the input features are divided into multiple equally sized windows, used for query (Q), key (K), and value (V) computation, respectively. Then, the lightweight convolutional network Fpred predicts the scale corresponding to each window. S i Sum and offset O i This is used to dynamically adjust the sampling positions of the key and value windows. Simultaneously, a wavelet transform path is connected in parallel within the module to decompose the input features into low-frequency components. yl and high frequency components yh The data is then reconstructed using inverse wavelet transform. Finally, the frequency path information is fused with the spatial attention result to achieve both spatial geometric adaptability and high-frequency information preservation.

[0102] Please see Figure 3 This figure illustrates the overall network architecture of this embodiment, DDFformer. The system consists of multiple layers, mainly including three stages: input feature extraction, two-domain alignment (FAB), and fusion and reconstruction. The input multispectral image (MS) and panchromatic image (PAN) are processed by stacked SFFA modules to extract preliminary features, which are then fed into the FAB module for spatial-frequency joint alignment. The aligned features are then fused with the MS features in the fusion module, and finally, a high-resolution image is output via the residual reconstruction module. The overall structure embodies the process logic of "feature extraction - feature alignment - feature fusion - residual reconstruction," emphasizing two-domain information fusion and adaptive perception.

[0103] Please see Figure 4 This figure illustrates the dual-domain feature alignment module (FAB) structure used in this embodiment. This module comprises three paths: spatial attention, frequency alignment, and modulation compensation, to achieve deep alignment of multimodal features. The spatial path generates query vectors from the PAN image and key-value pairs from the MS image, enhancing PAN features through cross-modal attention. The frequency path performs Fourier transforms on the PAN and MS features, reconstructing the frequency domain features by adding the amplitude and phase information. The modulation path generates modulation parameters γ and β based on the statistical characteristics of the MS, normalizing and recalibrating the PAN features to mitigate modal distribution differences. The three outputs are ultimately fused into aligned features, achieving joint alignment of PAN and MS in terms of spatial structure, frequency details, and distribution consistency.

[0104] Please see Figure 5This figure illustrates the fused images generated by different fusion methods (such as PNN, DiPAN, FAN, PMFT, and DDFformer in this embodiment), comparing their subjective visual quality. The results show that DDFformer can better restore details such as building edges and vegetation textures, with high visual sharpness, good spectral fidelity, and no obvious artifacts or aliasing issues. Overall, its subjective effect is better than other methods.

[0105] Please see Figure 6 This figure illustrates the objective evaluation metrics obtained by different methods on the WorldView-III remote sensing image dataset, including PSNR, SSIM, SAM, ERGAS, and QNR. The method in this embodiment meets or exceeds existing mainstream methods in all metrics, especially showing significant advantages in PSNR and SAM (reflecting spectral fidelity). This demonstrates the method's significant advantages in preserving image details and avoiding color deviation, validating its comprehensive improvement in fusion quality.

[0106] Please see Figure 7 This figure shows a residual heatmap between images generated by different methods and a reference image on the WorldView-III dataset. The residual map reflects the pixel error distribution between the predicted and ground truth images. It can be intuitively seen that the image generated by the DDFformer proposed in this embodiment has the smallest error with the real high-resolution image, and the residual map has almost no obvious structural difference, demonstrating better spatial detail restoration ability and spectral consistency.

[0107] Please see Figure 8 The figure illustrates the ablation experiment results for the two key modules, SFFA and FAB. The configurations are: no FAB and no SFFA, FAB only, SFFA only, and both. Comparison of PSNR, SSIM, SAM, and ERGAS metrics shows that both modules contribute positively to performance, with the most significant improvement observed when used together. This verifies the effectiveness and necessity of the dual-module structure in improving image fusion accuracy in this embodiment.

[0108] Compared to existing pancolor sharpening image fusion methods, the pancolor sharpening image fusion method based on a dual-domain flexible converter in this embodiment solves the following technical problems in image fusion, feature extraction, and information alignment in the prior art:

[0109] 1. This pancolor sharpening image fusion method addresses the problem of rigidly fixed attention windows in existing Transformer methods, which are ill-suited to complex terrain structures. Existing Transformer-based pancolor sharpening methods often employ fixed window sizes for self-attention calculations, lacking adaptability to image content and resulting in insufficient detail modeling capabilities for complex scenes such as terrain contours and building boundaries. This pancolor sharpening image fusion method, through a spatial-frequency flexible attention mechanism, dynamically adjusts the size and offset of the attention window based on image content, thereby better capturing feature information from irregular regions.

[0110] 2. This pancolor sharpening image fusion method addresses the issues of decoupling and insufficient information utilization in spatial and frequency domain modeling in existing methods. Traditional methods mostly model features only in the spatial domain or treat frequency information as an auxiliary module, failing to achieve joint spatial-frequency optimization. This pancolor sharpening image fusion method incorporates discrete wavelet transform and Fourier transform paths in both the feature extraction and fusion stages, effectively fusing high-frequency details and global structural information, thus improving the texture fidelity and edge sharpness of the fused image.

[0111] 3. This panchromatic sharpening image fusion method addresses the problem of large feature differences and a lack of effective alignment mechanisms between multi-source remote sensing images. PAN and MS images, originating from different sensors, exhibit significant modal differences, making direct fusion prone to introducing artifacts and color deviations. Existing methods often only consider spatial domain alignment, neglecting frequency domain feature differences. This panchromatic sharpening image fusion method constructs a "dual-domain feature alignment module," introducing contextual attention in the spatial domain and an FFT (Fast Fourier Transform) path in the frequency domain, significantly enhancing the consistency and fusionability of the two image features.

[0112] 4. This pancolor sharpening image fusion method solves the problems of high-frequency detail loss and edge blurring in the fused image. Since the self-attention mechanism is essentially a low-pass filter, it easily erases high-frequency texture information during image processing, leading to a blurry fusion result. This pancolor sharpening image fusion method introduces a wavelet decomposition path to specifically preserve high-frequency components and injects attention output in a learnable manner, effectively enhancing the detail representation of the fused image.

[0113] 5. This panchromatic sharpening image fusion method addresses the issue of insufficient generalization ability of existing panchromatic sharpening methods across different scenes. Existing methods often perform well on specific datasets, but their adaptability to diverse scenes such as complex terrain, urban buildings, and natural landscapes is poor. This panchromatic sharpening image fusion method employs a multi-scale modeling strategy, combined with hierarchical feature extraction and fusion design, enabling it to demonstrate good robustness and generalization on multiple real remote sensing datasets (such as WorldView-II, GaoFen-2, and WorldView-III).

[0114] In summary, this pancolor sharpening image fusion method based on a dual-domain flexible converter also has the following advantages:

[0115] First, this method introduces a spatial-frequency flexible attention mechanism (SFFA), breaking the fixed window limitation of traditional self-attention mechanisms in space. This enables adaptive perception and modeling of ground features, effectively adapting to complex situations in remote sensing images where ground features exhibit large scale variations and irregular shapes. Compared to fixed-window Transformers, it significantly reduces image structure distortion and detail blurring.

[0116] Secondly, this method connects high-frequency paths in parallel in the attention module, uses discrete wavelet transform (DWT) to extract high-frequency information of the image, and reconstructs it through inverse transform. This effectively compensates for the inherent low-pass characteristics of the Transformer attention mechanism, preserves high-frequency details such as texture and edges in the image, and improves image clarity and edge restoration capabilities.

[0117] Furthermore, the proposed dual-domain feature alignment simultaneously fuses PAN and MS features in both the spatial and frequency domains. It achieves explicit alignment of the amplitude and phase spectra through Fast Fourier Transform (FFT) and, combined with an instance normalization modulation strategy, compensates for distribution differences between modes, thereby improving fusion consistency and physical realism.

[0118] This method can achieve modular structure, flexible fusion method and high computational efficiency. It is compatible with existing CNN and Transformer architectures. It shows superior subjective and objective image quality indicators compared with existing technologies on multiple public remote sensing image fusion datasets, including PSNR, SSIM, SAM, ERGAS, QNR, etc. It has significant advantages in spectral fidelity and spatial clarity, and has strong engineering practical value and promotion potential.

[0119] Example 2

[0120] This embodiment provides a pancolor sharpening image fusion device based on a dual-domain flexible converter, which applies the pancolor sharpening image fusion method based on a dual-domain flexible converter described in Embodiment 1. The device includes a feature extraction module, a feature alignment module, a feature fusion module, a high-frequency reconstruction module, and an image generation module.

[0121] The feature extraction module performs multi-layer spatial-frequency joint attention operations on both the low-resolution multispectral image and the high-resolution panchromatic image to generate feature tensors MS and PAN. The feature alignment module performs dual-domain feature alignment on MS and PAN, aligning structural semantic features based on an attention mechanism. It fuses amplitude and phase spectral information through Fourier transform and matches modal distribution differences using instance normalization. The feature fusion module fuses the aligned features and MS at multiple levels, performing concatenation, convolution, and spatial-frequency joint attention operations at each level to obtain fused features. The high-frequency reconstruction module performs residual reconstruction on the fused features to generate a feature map with enhanced high-frequency details. The image generation module adds the reconstructed features element-wise to the upsampled low-resolution multispectral image to output a high-resolution multispectral image.

[0122] Example 3

[0123] This embodiment provides a computer terminal, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the pan-color sharpening image fusion method based on a dual-domain flexible converter as described in Embodiment 1.

[0124] The method in Example 1 can be applied in software form, such as by designing it as a standalone program and installing it on a computer terminal, which can be a computer, smartphone, control system, or other IoT device. Alternatively, the method in Example 1 can be designed as an embedded program and installed on a computer terminal, such as on a microcontroller.

[0125] Example 4

[0126] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the steps of the pan-color sharpening image fusion method based on a dual-domain flexible converter as described in Embodiment 1.

[0127] When applying the method of Example 1, it can be applied in the form of software, such as by designing it as a program that can run independently on a computer-readable storage medium. The computer-readable storage medium can be a USB flash drive, designed as a USB security token, and the program can be designed to start the entire method through an external trigger.

[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for fusing a full color sharpened image based on a dual domain flexible converter, characterized in that, It comprises the following steps: S1, feature extraction: performing multi-layer space-frequency joint attention operation on the low-resolution multispectral image and the high-resolution panchromatic image respectively to generate feature tensors MS and PAN; S2, feature alignment: performing double-domain feature alignment operation on MS and PAN, aligning structural semantic features based on attention mechanism, fusing amplitude and phase spectrum information through Fourier transform, and matching modal distribution differences through instance normalization; the double-domain feature alignment is realized through a space attention path and a frequency alignment path, combined with a modulation compensation path, and the output results of the three paths are fused into aligned features; wherein the space path generates a query vector from the PAN image and a key-value pair from the MS image; the frequency path performs Fourier transform on PAN and MS, adds the amplitude and phase information, and reconstructs the frequency domain features; the modulation path generates modulation parameters according to the statistical characteristics of MS, and normalizes and recalibrates the PAN features; S3, feature fusion: performing multi-level fusion of the aligned features and MS, each level sequentially performing splicing, convolution and space-frequency joint attention operation to obtain fused features; S4, high-frequency reconstruction: performing residual reconstruction on the fused features to generate a feature map with enhanced high-frequency details; S5, image generation: element-wise adding the reconstructed features to the up-sampled low-resolution multispectral image to output a high-resolution multispectral image.

2. The dual domain flexible converter based panchromatic sharpening image fusion method of claim 1, wherein, The space-frequency joint attention operation is realized by the following methods: (1) predicting the sampling scale of each window based on the input features S i and offset O i to generate adaptive sampling coordinates; (2) sampling and projecting from the key / value tensor, combined with the query vector Q computing the attention output; (3) the output result and the high-frequency features extracted by wavelet transform F frequency weighted fusion.

3. The dual domain flexible converter based panchromatic sharpening image fusion method of claim 2, wherein, The calculation formula of the adaptive sampling coordinates is: wherein p sample denotes the adaptive sampling coordinate, p base denotes the reference grid coordinate, p ref denotes the reference vector.

4. The dual domain flexible converter based panchromatic sharpening image fusion method of claim 3, wherein, The feature calculation formula of the space alignment is: wherein, F SIM denotes a spatially aligned feature, K denotes a key matrix, V denotes a value matrix, d k denotes the dimension of the key / query vector.

5. The dual domain flexible converter based panchromatic sharpening image fusion method of claim 4, wherein, The fused feature calculation formula is: wherein, F SFA denotes a linear projection of the fusion feature, Kd denotes a linear projection of the K denotes a linear projection of the Vd denotes a linear projection of the V denotes a linear projection of the B denotes a position bias term, and λ denotes a learnable fusion weight.

6. The dual domain flexible converter based panchromatic sharpening image fusion method of claim 1, wherein, In step S2, the Fourier transform fusion method comprises the following steps: (1) Fourier transform MS and PAN to obtain corresponding complex spectra: wherein F () denotes a fast Fourier transform, A () denotes a magnitude spectrum, (2) Weighted fusion or direct addition of the amplitude spectrum and the phase spectrum of the complex spectrum to obtain a comprehensive frequency domain: () denotes a phase spectrum, j denotes the imaginary unit; (3) Re-combine the fused amplitude and phase into a complex spectrum, and then perform inverse Fourier transform to obtain the frequency domain aligned features: wherein A Distribute the calibration through the modulation mechanism, and obtain the mean and variance of MS through channel statistics. Predict the modulation parameters after splicing MS and PAN to reconstruct PAN after instance normalization; wherein the formula for instance-normalized PAN is: represents the amplitude spectrum fusion, σ μ represents the phase spectrum fusion;​​ γ wherein F FIM represents the frequency domain aligned feature, represents a take real operation, F -1 () represents an inverse Fourier transform.

7. The dual domain flexible converter based panchromatic sharpening image fusion method of claim 6, wherein, IN wherein F Mod denotes the distribution calibrated feature, The aligned features are represented as: 2 denotes the variance, ГK denotes the mean, MS , β denotes the modulation parameter, The application is as described in any one of claims 1-9; the device comprises: () denotes instance normalization.

8. The dual domain flexible converter based panchromatic sharpening image fusion method of claim 7, wherein, A feature extraction module for performing multi-layer space-frequency joint attention operation on the low-resolution multispectral image and the high-resolution panchromatic image respectively to generate feature tensors MS and PAN; In the formulae, F AB denotes the alignment feature.

9. The dual domain flexible converter based panchromatic sharpening image fusion method of claim 8, wherein, In step S5, first, hierarchical fusion is performed using a stacked structure to obtain F AB The MS is input into the fusion network, and a predicted image is output through a residual module, and then added to the MS image after bicubic interpolation to output the high-resolution multispectral image; wherein the calculation formula of the high-resolution multispectral image is: In the formula, R () represents a residual module operation, A feature alignment module for performing double-domain feature alignment operation on MS and PAN, aligning structural semantic features based on attention mechanism, fusing amplitude and phase spectrum information through Fourier transform, and matching modal distribution differences through instance normalization; () represents the fusion network, A feature fusion module for performing multi-level fusion of the aligned features and MS, each level sequentially performing splicing, convolution and space-frequency joint attention operation to obtain fused features; bic represents a low-resolution multispectral image after upsampling.

10. A full color sharpening image fusion device based on dual domain flexible converter, characterized in that, ​ ​ ​ ​ a high-frequency reconstruction module configured to perform residual reconstruction on the fused features to generate a feature map with high-frequency detail enhancement; an image generation module configured to element-wise add the reconstructed features and the up-sampled low-resolution multispectral image to output a high-resolution multispectral image.

Images