Panchromatic sharpening image fusion method and device based on double-domain flexible converter

Abstract

The invention discloses a panchromatic sharpening image fusion method and device based on a double-domain flexible converter. The method comprises the following steps: performing multilayer space-frequency joint attention operation on a low-resolution multispectral image and a high-resolution panchromatic image to generate a feature tensor; performing double-domain feature alignment operation, aligning structural semantic features based on an attention mechanism, fusing amplitude and phase frequency spectrum information through Fourier transform, and matching modal distribution differences by using instance normalization; multi-level fusion is carried out, and splicing, convolution and space-frequency joint attention operation are carried out on each level in sequence; carrying out residual error reconstruction on the fusion features; and adding the reconstructed features and the up-sampled low-resolution multispectral image element by element, and outputting a high-resolution multispectral image. According to the method, image structure distortion and detail blurring can be remarkably reduced, the image definition and the edge reduction capability are improved, the fusion consistency and the physical authenticity are improved, and the detail expressive force of the fused image is enhanced.

Description

Technical Field

[0001] The present invention relates to a full-color sharpening image fusion method in the field of image processing technology, in particular to a full-color sharpening image fusion method based on a dual-domain flexible converter, and also to a full-color sharpening image fusion device based on a dual-domain flexible converter. Background Art

[0002] Pansharpening is a key task in remote sensing image processing. It aims to fuse low-resolution multispectral (LRMS) images with high-resolution panchromatic (PAN) images to generate high-resolution multispectral (HRMS) images. This technique is of great significance in remote sensing applications such as land use classification, urban planning, and environmental monitoring. Existing pansharpening methods can be mainly divided into two categories: traditional model-driven methods and data-driven methods. Traditional methods include the IHS (Intensity-Hue-Saturation) transform [Carper et al., 1990], the Brovey transform [Gillespie, 1987], and the principal component analysis (PCA) method [Aiazzi et al., 2007]. While these methods are computationally efficient, they are prone to spectral distortion and spatial artifacts in the fusion results.

[0003] Convolutional neural networks (CNN) are difficult to capture long-range dependencies in images due to limited receptive fields, resulting in insufficient structure restoration in generated images. Models based on the Transformer structure use self-attention mechanisms to model long-range dependencies, improving the ability to restore spatial details. However, the current mainstream Transformer methods still have the following significant problems: (1) Attention window is rigidly fixed: existing methods mostly use fixed-size sliding windows for attention calculation, which is difficult to adapt to complex and variable ground feature forms in remote sensing images, and is prone to cause local structure information loss and edge blurring; (2) Unable to jointly model spatial and frequency information: Most methods only model features in the spatial domain, ignoring the importance of the frequency domain (such as texture and high-frequency details), or treating frequency processing as an independent module rather than integrating it as a whole, resulting in insufficient utilization of spectral information and loss of details in the fusion result; (3) Insufficient alignment of modalities: PAN and MS images are derived from different sensors, and there are significant modal differences. If there is no effective alignment mechanism, direct fusion may cause artifacts or color deviations, affecting image quality. Methods that model both spatial and frequency information can help improve image fusion performance, but they are mainly aimed at hyperspectral images and lack in-depth research on feature alignment and window adaptation mechanisms in PAN-MS image fusion.

[0004] Therefore, the existing panchromatic sharpening method has the problems of being difficult to adapt to complex ground feature structures, large feature differences between multi-source remote sensing images, and lack of effective alignment mechanism. SUMMARY

[0005] To solve the technical problems of existing panchromatic sharpening methods being difficult to adapt to complex ground feature structures, large feature differences between multi-source remote sensing images, and lack of effective alignment mechanism, the present application provides a panchromatic sharpening image fusion method based on a dual-domain flexible converter and a device thereof.

[0006] The present application adopts the following technical solutions: a panchromatic sharpening image fusion method based on a dual-domain flexible converter, comprising the following steps: S1, feature extraction: performing multi-layer spatial-frequency joint attention operation on low-resolution multispectral images and high-resolution panchromatic images respectively to generate feature tensors MS and PAN; S2, feature alignment: performing dual-domain feature alignment operation on MS and PAN, aligning structural semantic features based on attention mechanism, fusing amplitude and phase spectral information through Fourier transform, and matching modal distribution differences using instance normalization; S3, feature fusion: performing multi-level fusion on the aligned features and MS, performing splicing, convolution, and spatial-frequency joint attention operation at each level in turn to obtain fusion features; S4, high-frequency reconstruction: performing residual reconstruction on the fusion features to generate a feature map with enhanced high-frequency details. S5, image generation: add the reconstructed features to the up-sampled low-resolution multispectral images element by element, and output high-resolution multispectral images.

[0007] The application can dynamically adjust the size and offset of the attention window according to the image content through the space-frequency flexible attention mechanism, thereby better capturing the feature information of irregular areas, and by constructing a "double-domain feature alignment module", respectively introducing context attention in the spatial domain and introducing a Fourier transform path in the frequency domain, the consistency and fusibility of the two image features are significantly enhanced, thereby solving the technical problems that the existing panchromatic sharpening method is difficult to adapt to complex ground object structures, and there is a large feature difference between multi-source remote sensing images and a lack of effective alignment mechanism.

[0008] As a further improvement of the above scheme, the space-frequency joint attention operation is realized by the following ways: (1) predicting the sampling scale of each window based on the input features S i and offset O i to generate adaptive sampling coordinates; (2) sampling and projecting from the key / value tensor, combined with the query vector Q to calculate the attention output; (3) weighted fusion of the output result and the high-frequency features extracted by wavelet transform F frequency

[0009] Further, the calculation formula of the adaptive sampling coordinates is:

[0010] In the formula, p sample wherein the adaptive sampling coordinates are represented by p base the reference vector is represented by p ref

[0011] Further, the feature calculation formula of spatial alignment is:

[0012] In the formula, F SIM wherein the spatially aligned features are represented by K the key matrix is represented by V the value matrix is represented by d k the dimension of the key / query vector is represented by

[0013] ​​Further, the fusion feature calculation formula is:

[0014] wherein, F SFA denotes the fusion feature, Kd denotes a linear projection on K , Vd denotes a linear projection on V , B denotes a position bias term, and λ denotes a learnable fusion weight.

[0015] Further, in step S2, the Fourier transform fusion method comprises the following steps: (1) Fourier transform is performed on the MS and the PAN to obtain corresponding complex spectra:

[0016] wherein, F denotes a fast Fourier transform, A denotes an amplitude spectrum, θ denotes a phase spectrum, j denotes an imaginary unit; (2) the amplitude spectrum and the phase spectrum of the complex spectrum are respectively weighted fused or directly added to obtain a comprehensive frequency domain:

[0017] wherein, A denotes amplitude spectrum fusion, fused denotes phase spectrum fusion; θ (3) the fused amplitude and phase are recombined into a complex spectrum, and then inverse Fourier transform is performed to obtain a frequency domain aligned feature: fused

[0018] wherein, F FIM denotes the frequency domain aligned feature, denotes a real part operation, F -1 denotes an inverse Fourier transform.

[0019] Further, distribution calibration is performed through a modulation mechanism, the mean and variance of the MS are obtained through channel statistics, the modulation parameters are predicted after the MS and the PAN are spliced, and the PAN is reconstructed after instance normalization; wherein, the formula for instance normalization reconstruction of the PAN is:

[0020] ​​wherein, F Mod denotes the distribution calibrated feature, and sigma 2 denotes variance, μ denotes mean, γ , β denotes modulation parameter, IN denotes instance normalization.

[0021] Further, the alignment feature is represented as:

[0022] wherein, F AB denotes the alignment feature.

[0023] Further, in step S5, hierarchical fusion is first performed by using a stacked structure to fuse F AB and the MS is input into the fusion network together, and a predicted image is output through a residual module, and then the predicted image is added to the MS image after bicubic interpolation to output the high-resolution multispectral image; wherein the calculation formula of the high-resolution multispectral image is:

[0024] wherein, R denotes residual module operation, denotes the fusion network, MS bic denotes the up-sampled low-resolution multispectral image.

[0025] The application also provides a panchromatic sharpening image fusion device based on a dual-domain flexible converter, which applies any of the panchromatic sharpening image fusion methods based on the dual-domain flexible converter described above; the device comprises: a feature extraction module for performing multi-layer space-frequency joint attention operation on the low-resolution multispectral image and the high-resolution panchromatic image respectively to generate feature tensors MS and PAN; a feature alignment module for performing dual-domain feature alignment operation on MS and PAN, aligning structure semantic features based on an attention mechanism, fusing amplitude and phase spectral information through Fourier transform, and matching modal distribution differences by using instance normalization; a feature fusion module for performing multi-level fusion on the alignment feature and MS, and sequentially performing splicing, convolution and space-frequency joint attention operation at each level to obtain a fusion feature; a high-frequency reconstruction module for performing residual reconstruction on the fusion feature to generate a feature map with enhanced high-frequency details; an image generation module configured to add the reconstructed features to the upsampled low-resolution multispectral image element by element, outputting a high-resolution multispectral image.

[0026] Compared with the existing panchromatic sharpening method, the panchromatic sharpening image fusion method based on the dual-domain flexible converter and the device have the following beneficial effects: 1. The panchromatic sharpening image fusion method based on the dual-domain flexible converter solves the problem of rigid fixed attention window in the existing Transformer method, which is difficult to adapt to complex ground feature structures. The existing panchromatic sharpening method based on Transformer mostly uses fixed window size for self-attention calculation, lacks adaptive ability to image content, and leads to insufficient detail modeling ability for complex scenes such as terrain contour and building boundary. The panchromatic sharpening image fusion method breaks the fixed window limit of traditional self-attention mechanism in space through a space-frequency flexible attention mechanism, realizes adaptive perception and modeling of ground feature structures, and can dynamically adjust the size and offset of the attention window according to the image content. It can effectively adapt to the complex situation of large ground feature scale change and irregular shape in remote sensing images, better capture feature information in irregular areas, and significantly reduce image structure distortion and detail blur compared with fixed window Transformer.

[0027] 2. The panchromatic sharpening image fusion method based on the dual-domain flexible converter solves the problem of decoupling in spatial and frequency domain modeling and insufficient information utilization in the existing method. Most traditional methods only model features in the spatial domain or process frequency information as an auxiliary module, and cannot realize spatial-frequency joint optimization. The panchromatic sharpening image fusion method introduces discrete wavelet transform and Fourier transform paths in the feature extraction and fusion stage, effectively fuses high-frequency details and global structure information, and improves the texture fidelity and edge clarity of the fused image. Through inverse transform reconstruction, the natural low-pass characteristic of the Transformer attention mechanism is effectively compensated, the texture, edge and other high-frequency details in the image are preserved, and the image clarity and edge restoration ability are improved.

[0028] 3. The panchromatic sharpening image fusion method based on the dual-domain flexible converter solves the problem of large feature difference between multi-source remote sensing images and the lack of effective alignment mechanism. PAN images and MS images have large modal differences due to different sensors, and direct fusion can easily introduce artifacts and color deviations. Existing methods mostly only consider spatial domain alignment, ignoring the feature difference in the frequency domain. The panchromatic sharpening image fusion method constructs a "dual-domain feature alignment module" to introduce context attention in the spatial domain and an FFT (Fast Fourier Transform) path in the frequency domain, significantly enhancing the consistency and fusibility of the features of the two images. Moreover, explicit alignment of the amplitude spectrum and the phase spectrum is achieved through Fast Fourier Transform (FFT), and the distribution difference between modalities is compensated by combining the instance normalization modulation strategy, thereby improving the consistency and physical authenticity of the fusion.

[0029] 4. The panchromatic sharpening image fusion method based on the dual-domain flexible converter solves the problem of high-frequency detail loss and edge blur after fusion. Since the self-attention mechanism is essentially a low-pass filter, it can easily erase high-frequency texture information when processing images, resulting in blurred fusion results. The panchromatic sharpening image fusion method introduces a wavelet decomposition path to specifically preserve high-frequency components and inject attention output in a learnable manner, effectively enhancing the detail expressiveness of the fusion image.

[0030] 5. The panchromatic sharpening image fusion method based on the dual-domain flexible converter solves the problem of insufficient generalization ability of existing panchromatic sharpening methods in different scenes. Existing methods often perform well on specific datasets, but have poor adaptability to diverse scenes such as complex terrain, urban buildings, and natural landscapes. The panchromatic sharpening image fusion method uses a multi-scale modeling strategy combined with hierarchical feature extraction and fusion design, making it robust and generalizable on multiple real remote sensing datasets such as WorldView-II, GaoFen-2, and WorldView-III. Moreover, this method can achieve structural modularity, flexible fusion, high computational efficiency, and compatibility with existing CNN and Transformer architectures, outperforming existing technologies in terms of objective and subjective image quality indicators such as PSNR, SSIM, SAM, ERGAS, QNR, especially in spectral fidelity and spatial clarity. It has strong engineering practical value and promotion potential.

[0031] 6. The panchromatic sharpening image fusion device based on the dual-domain flexible converter has the same beneficial effects as the panchromatic sharpening image fusion method described above, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1Flow chart of the panchromatic sharpening image fusion method based on dual-domain flexible converter of embodiment 1 of the present application.

[0033] Figure 2 Internal structure diagram of a spatial-frequency attention module used in the panchromatic sharpening image fusion method of embodiment 1 of the present application.

[0034] Figure 3 Structure diagram of an overall network architecture used in the panchromatic sharpening image fusion method of embodiment 1 of the present application. DDFformer

[0035] Figure 4 Structure diagram of a dual-domain feature alignment module (FAB) used in the panchromatic sharpening image fusion method of embodiment 1 of the present application.

[0036] Figure 5 Comparison diagram of the panchromatic sharpening image fusion method of embodiment 1 of the present application and fusion images generated by different fusion methods.

[0037] Figure 6 Data comparison diagram of objective evaluation indexes obtained by the panchromatic sharpening image fusion method of embodiment 1 of the present application and different methods on a WorldView-III remote sensing image dataset.

[0038] Figure 7 Residual map heat map between images generated by different methods and a reference image on a WorldView-III dataset in embodiment 1 of the present application.

[0039] Figure 8 Effect diagram of the ablation experiment on the two key modules SFFA and FAB in the panchromatic sharpening image fusion method of embodiment 1 of the present application. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present application clearer and more comprehensible, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0041] Embodiment 1 Please refer to Figure 1 The present embodiment provides a panchromatic sharpening image fusion method based on a dual-domain flexible converter, which aims to fuse a low-resolution multi-spectral image (LRMS) and a high-resolution panchromatic image (PAN) to generate a high-resolution multi-spectral image (HRMS). In the present embodiment, the panchromatic sharpening image fusion method includes the following steps (S1-S5).

[0042] ​S1, feature extraction: multi-layer space-frequency joint attention operation is performed on the low-resolution multispectral image and the high-resolution panchromatic image respectively to generate feature tensors MS and PAN. In this embodiment, a space-frequency flexible attention module (SFFA) is introduced to realize adaptive spatial perception through a flexible window mechanism.

[0043] S2, feature alignment: double-domain feature alignment operation is performed on MS and PAN, and structure semantic features are aligned based on attention mechanism. The amplitude and phase spectrum information are fused through Fourier transform, and the modal distribution difference is matched by instance normalization. In this embodiment, the MS and PAN modal features are aligned in space and frequency in a double-path way. In the spatial path, the input features are completed by attention mechanism, and the process is as follows: a query Q=Conv(PAN) is constructed, and the key and value are extracted from the MS feature.

[0044] In this embodiment, the feature calculation formula of spatial alignment is:

[0045] In the formula, F SIM represents the feature of spatial alignment, K represents the key matrix, V represents the value matrix, d k represents the dimension of the key / query vector.

[0046] In this embodiment, in the frequency path, the system uses fast Fourier transform (FFT) to extract frequency features from the high-resolution panchromatic image (PAN) and the low-resolution multispectral image (MS). Specifically, the Fourier transform fusion method includes the following steps: (1) Fourier transform is performed on MS and PAN to obtain corresponding complex spectrum:

[0047] In the formula, F (1) represents fast Fourier transform, A (2) represents amplitude spectrum, θ (3) represents phase spectrum, j represents the imaginary unit; (2) The amplitude spectrum and the phase spectrum of the complex spectrum are weighted and fused or directly added to obtain the comprehensive frequency domain:

[0048] In the formula, A (1) represents amplitude spectrum fusion, fused (2) represents phase spectrum fusion; θ fused ​​(3) The combined amplitude and phase are recombined into a complex spectrum, and the frequency domain alignment feature is obtained by inverse Fourier transform:

[0049] In the formula, F FIM The frequency domain alignment feature is represented by The real part operation is represented by F -1 The inverse Fourier transform is represented by. This frequency alignment path can effectively fuse the frequency component information from MS and PAN, especially in the high-frequency regions such as edges and textures, showing better alignment effect, which helps to enhance the clarity and structural consistency of the final fused image.

[0050] S3, feature fusion: multi-level fusion of aligned features and MS, each level sequentially performing stitching, convolution and spatial-frequency joint attention operation to obtain fused features.

[0051] The traditional Transformer fixed window cannot capture irregular features. In this embodiment, a lightweight network Fpred is used to realize spatial-frequency joint attention operation, mainly through the following ways: (1) Based on the input feature, the sampling scale of each window is predicted S i and offset O i to generate adaptive sampling coordinates; (2) Sampling and projection from the key / value tensor, combined with the query vector Q to calculate the attention output; (3) Weighted fusion of the output result and the high-frequency feature extracted by wavelet transform (such as Haar wavelet, or other wavelets) F frequency .

[0052] The calculation formula of the adaptive sampling coordinates is:

[0053] In the formula, p sample The adaptive sampling coordinates are represented by p base The reference vector is represented by p ref

[0054] The final fused feature is obtained, and the calculation formula is:

[0055] In the formula, F SFA The fused feature is represented by Kd The operation is represented by​K The linear projection of Vd Express V The linear projection of B represents the position bias term, and λ represents the learnable fusion weight.

[0056] S4. High-frequency reconstruction: Residual reconstruction is performed on the fused features to generate a feature map with enhanced high-frequency details. In this embodiment, to further reduce modal differences, distribution calibration is performed through the modulation mechanism, and the mean and variance of the MS are obtained through channel statistics. The modulation parameters are predicted after splicing the MS and PAN, and the PAN is reconstructed after instance normalization. The formula for reconstructing the PAN through instance normalization is:

[0057] Where, F Mod Represents the distribution calibration characteristics, σ 2 represents the variance, μ represents the mean, γ 、 β represents the modulation parameters, IN () indicates instance normalization.

[0058] Therefore, the final output alignment features are:

[0059] Where, F AB Represents an alignment feature.

[0060] S5, image generation: add the reconstructed features to the upsampled low-resolution multispectral image element by element to output a high-resolution multispectral image. In this embodiment, a stacked structure is first used for hierarchical fusion to F AB The MS is input into the fusion network together, and the predicted image is output through the residual module. It is then added to the MS image after bicubic interpolation to output a high-resolution multispectral image. This fusion strategy enables the system to have structural adaptability in both spatial and frequency domains, accurately restore the edges and spectral details of the ground objects, and demonstrate excellent subjective and objective evaluation performance in actual measurements. The calculation formula for the high-resolution multispectral image is:

[0061] Where, R () represents the residual module operation, represents the fusion network, MS bic Represents the upsampled low-resolution multispectral image.

[0062] See also Figure 2, which shows the internal structure of a spatial-frequency attention module, i.e., to implement the spatial-frequency joint attention operation. First, the input features are divided into multiple equal-sized windows, which are used for the calculation of queries ( Q ), keys ( K ) and values ( V ), respectively. Then, the light convolutional network Fpred predicts the scale S i and the offset O i for each window, which are used to dynamically adjust the sampling positions of the key and value windows. At the same time, a wavelet transform path is also connected in parallel in the module, which decomposes the input features into low-frequency components yl and high-frequency components yh , and reconstructs them through inverse wavelet transform. Finally, the frequency path information is fused with the spatial attention result to realize the dual ability of spatial geometric adaptability and high-frequency information preservation.

[0063] Please refer to Figure 3 , which shows the overall network architecture of the embodiment DDFformer . The system is composed of multiple levels, mainly including: input feature extraction, dual-domain alignment (FAB), fusion and reconstruction three stages. The input multispectral image (MS) and panchromatic image (PAN) are respectively extracted through the stacked SFFA module, and then sent to the FAB module to realize the spatial-frequency joint alignment. The aligned features are fused with the MS features in the fusion module, and finally output the high-resolution image through the residual reconstruction module. The overall structure embodies the process logic of "feature extraction-feature alignment-feature fusion-residual reconstruction", emphasizing the fusion of dual-domain information and adaptive perception.

[0064] Please refer to Figure 4 , which shows that the embodiment can utilize the dual-domain feature alignment module (FAB) structure, which includes three paths of spatial attention, frequency alignment and modulation compensation, for realizing the deep alignment of multi-modal features. The spatial path generates query vectors from PAN images and key-value pairs from MS images, and enhances PAN features through cross-modal attention. The frequency path performs Fourier transform on PAN and MS features, adds the amplitude and phase information, and reconstructs the frequency domain features. The modulation path generates modulation parameters γ and β according to the statistical characteristics of MS, normalizes and re-scales the PAN features, and alleviates the modal distribution difference. The three outputs are finally fused into aligned features, realizing the joint alignment of PAN and MS from the aspects of spatial structure, frequency details and distribution consistency.

[0065] Please refer to Figure 5The figure shows the fused images generated by different fusion methods (such as PNN, DiPAN, FAN, PMFT, and DDFformer of the embodiment), and compares their subjective visual quality. The results show that DDFformer can better restore architectural edges, vegetation texture and other details, has high visual sharpness, good spectral fidelity, and no obvious artifacts or aliasing problems, and the overall subjective effect is better than other methods.

[0066] Referring to Figure 6 The figure shows the objective evaluation indicators obtained by different methods on the WorldView-III remote sensing image dataset, including PSNR, SSIM, SAM, ERGAS and QNR indicators. The method of the embodiment achieves or exceeds the existing mainstream methods in all indicators, especially in PSNR and SAM (reflecting spectral fidelity), which shows that the method has significant advantages in preserving image details and avoiding color deviation, and verifies the overall improvement of its fusion quality.

[0067] Referring to Figure 7 The figure shows the residual map heat map between the images generated by different methods and the reference image on the WorldView-III dataset. The residual map reflects the pixel error distribution between the predicted image and the true image. It can be seen directly that the error between the image generated by DDFformer proposed in the embodiment and the true high-resolution image is the smallest, and the residual map has almost no obvious structural difference, showing better spatial detail restoration ability and spectral consistency.

[0068] Referring to Figure 8 The figure shows the ablation experiment results of the two key modules SFFA and FAB. The configurations are: no FAB and no SFFA, only FAB, only SFFA, and both. By comparing the PSNR, SSIM, SAM and ERGAS indicators, it can be seen that both modules positively contribute to performance, and the improvement is most significant when used together, verifying the effectiveness and necessity of the dual-module structure of the embodiment in image fusion accuracy.

[0069] Compared with existing panchromatic sharpening image fusion methods, the panchromatic sharpening image fusion method based on the dual-domain flexible converter of the embodiment solves the following technical problems in the prior art in image fusion, feature extraction and information alignment: 1、The panchromatic sharpening image fusion method solves the problem of rigid fixed attention window in existing Transformer methods, which is difficult to adapt to complex feature structures. Existing panchromatic sharpening methods based on Transformer mostly use fixed window size for self-attention calculation, lacking the ability to adapt to image content, resulting in insufficient detail modeling capability for complex scenes such as terrain contours and building boundaries. The panchromatic sharpening image fusion method can dynamically adjust the size and offset of the attention window according to the image content through the space-frequency flexible attention mechanism, thus better capturing feature information in irregular areas.

[0070] 2、The panchromatic sharpening image fusion method solves the problem of decoupling in spatial and frequency domain modeling and insufficient information utilization in existing methods. Most traditional methods only model features in the spatial domain or process frequency information as auxiliary modules, failing to achieve spatial-frequency joint optimization. The panchromatic sharpening image fusion method introduces discrete wavelet transform and Fourier transform paths in the feature extraction and fusion stage, effectively fusing high-frequency details and global structure information, and improving the texture fidelity and edge clarity of the fused image.

[0071] 3、The panchromatic sharpening image fusion method solves the problem of large feature difference between multi-source remote sensing images and lack of effective alignment mechanism. PAN images and MS images come from different sensors, resulting in large modal differences, and direct fusion may introduce artifacts and color deviations. Existing methods mostly only consider spatial domain alignment, ignoring frequency domain feature differences. The panchromatic sharpening image fusion method constructs a "double-domain feature alignment module" to introduce context attention in the spatial domain and FFT (Fast Fourier Transform) path in the frequency domain, significantly enhancing the consistency and fusibility of the features of the two images.

[0072] 4、The panchromatic sharpening image fusion method solves the problem of high-frequency detail loss and edge blur after fusion. Since the self-attention mechanism is essentially a low-pass filter, it can easily erase high-frequency texture information when processing images, resulting in blurred fusion results. The panchromatic sharpening image fusion method introduces wavelet decomposition paths to specifically preserve high-frequency components and inject attention outputs in a learnable manner, effectively enhancing the detail expressiveness of the fused image.

[0073] 5、The panchromatic sharpening image fusion method solves the problem of insufficient generalization ability of existing panchromatic sharpening methods in different scenes. Existing methods often perform well on specific datasets, but have poor adaptability to diverse scenes such as complex terrain, urban buildings, and natural landscapes. The panchromatic sharpening image fusion method adopts a multi-scale modeling strategy, combining hierarchical feature extraction and fusion design, making it perform well in multiple real remote sensing datasets (such as WorldView-II, GaoFen-2, and WorldView-III).

[0074] In summary, the panchromatic sharpening image fusion method based on the dual-domain flexible converter also has the following advantages: First, the method breaks the fixed window limit of the traditional self-attention mechanism in space by introducing a spatial-frequency flexible attention mechanism (SFFA), achieving adaptive perception and modeling of ground object structures. It can effectively adapt to the complex situation of large scale changes and irregular shapes of ground objects in remote sensing images. Compared with the fixed window Transformer, it can significantly reduce the image structure distortion and detail blur problems.

[0075] Second, the method parallelizes the high-frequency path in the attention module, uses discrete wavelet transform (DWT) to extract the high-frequency information of the image, and reconstructs through inverse transform, effectively compensating for the natural low-pass characteristics of the Transformer attention mechanism, preserving the texture, edges and other high-frequency details in the image, and improving the image clarity and edge restoration ability.

[0076] Third, the dual-domain feature alignment proposed by the method simultaneously fuses PAN and MS features in the spatial and frequency domains, realizes explicit alignment of amplitude spectrum and phase spectrum through fast Fourier transform (FFT), and combines instance normalization modulation strategy to compensate for the distribution difference between modalities, thereby improving the fusion consistency and physical authenticity.

[0077] The method can realize structural modularization, flexible fusion, high computational efficiency, and compatibility with existing CNN and Transformer architectures. It outperforms existing technologies in terms of objective and subjective image quality indicators, including PSNR, SSIM, SAM, ERGAS, QNR, etc., especially in spectral fidelity and spatial clarity. It has strong engineering practical value and promotion potential.

[0078] Embodiment 2 The embodiment provides a panchromatic sharpening image fusion device based on a dual-domain flexible converter. The device applies the panchromatic sharpening image fusion method based on the dual-domain flexible converter in embodiment 1. The device comprises a feature extraction module, a feature alignment module, a feature fusion module, a high-frequency reconstruction module, and an image generation module.

[0079] The feature extraction module is configured to perform a multi-layer spatial-frequency joint attention operation on the low-resolution multispectral image and the high-resolution panchromatic image respectively to generate feature tensors MS and PAN. The feature alignment module is configured to perform a dual-domain feature alignment operation on MS and PAN, align structural semantic features based on an attention mechanism, fuse amplitude and phase spectral information through Fourier transform, and match modal distribution differences using instance normalization. The feature fusion module is configured to perform multi-level fusion on the aligned features and MS, with each level sequentially performing splicing, convolution, and spatial-frequency joint attention operation to obtain fused features. The high-frequency reconstruction module is configured to perform residual reconstruction on the fused features to generate a feature map with enhanced high-frequency details. The image generation module is configured to add the reconstructed features and the up-sampled low-resolution multispectral image element by element to output a high-resolution multispectral image.

[0080] Embodiment 3 The embodiment provides a computer terminal including a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor implements the steps of the dual-domain flexible converter-based panchromatic sharpening image fusion method of embodiment 1 when executing the program.

[0081] The method of embodiment 1 can be applied in the form of software, such as a program designed to run independently, installed on a computer terminal, which can be a computer, a smartphone, a control system, and other Internet of Things devices, etc. The method of embodiment 1 can also be designed as an embedded program installed on a computer terminal, such as a single-chip microcomputer.

[0082] Embodiment 4 The embodiment provides a computer-readable storage medium having a computer program stored thereon. The program, when executed by a processor, implements the steps of the dual-domain flexible converter-based panchromatic sharpening image fusion method of embodiment 1.

[0083] The method of embodiment 1 can be applied in the form of software, such as a program designed to run independently on a computer-readable storage medium, which can be a U disk, designed as a U disk, and designed as a program that starts the entire method through external triggering through a U disk.

[0084] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A full-color sharpening image fusion method based on a dual-domain flexible converter, characterized in that: It includes the following steps: S1. Feature extraction: Perform multi-layer spatial-frequency joint attention operations on low-resolution multispectral images and high-resolution panchromatic images to generate feature tensors MS and PAN; S2. Feature alignment: Perform dual-domain feature alignment on MS and PAN, align structural semantic features based on the attention mechanism, fuse amplitude and phase spectrum information through Fourier transform, and use instance normalization to match modal distribution differences; S3, feature fusion: The aligned features and MS are fused at multiple levels, with each level performing concatenation, convolution, and spatial-frequency joint attention operations in turn to obtain fused features; S4, high frequency reconstruction: performing residual reconstruction on the fused features to generate a feature map with enhanced high frequency details; S5. Image generation: Add the reconstructed features to the upsampled low-resolution multispectral image element by element to output a high-resolution multispectral image.

2. The method for full-color sharpening image fusion based on dual-domain flexible converter according to claim 1, wherein: The spatial-frequency joint attention operation is implemented as follows: (1) Predict the sampling scale of each window based on the input features S i and offset O i , generate adaptive sampling coordinates; (2) Sample and project from the key / value tensor, combined with the query vector Q Compute attention output; (3) Compare the output results with the high-frequency features extracted by wavelet transform F frequency Weighted fusion.

3. The method for full-color sharpening image fusion based on dual-domain flexible converter according to claim 2, wherein: The calculation formula of the adaptive sampling coordinates is: ， Where, p sample represents the adaptive sampling coordinates, p base represents the reference grid coordinates, p ref represents the reference vector.

4. The method for full-color sharpening image fusion based on dual-domain flexible converter according to claim 3, wherein: The feature calculation formula for spatial alignment is: ， Where, F SIM Features representing spatial alignment, K represents the bond matrix, V represents the value matrix, d k Indicates the dimensions of the key / query vector.

5. The method for full-color sharpening image fusion based on dual-domain flexible converter according to claim 4, wherein: The fusion feature calculation formula is: ， Where, F SFA represents the fusion feature, Kd Express K The linear projection of Vd Express V The linear projection of B represents the position bias term, and λ represents the learnable fusion weight.

6. The method for full-color sharpening image fusion based on dual-domain flexible converter according to claim 1, wherein: In step S2, the Fourier transform fusion method includes the following steps: (1) Perform Fourier transform on MS and PAN to obtain the corresponding complex spectrum: ， Where, F () represents fast Fourier transform, A () represents the amplitude spectrum, θ () represents the phase spectrum, j represents an imaginary unit; (2) The amplitude spectrum and phase spectrum of the complex spectrum are weighted fused or directly added to obtain the comprehensive frequency domain: , Where, A ( fused ) represents amplitude spectrum fusion, θ ( fused ) represents phase spectrum fusion; (3) First, the fused amplitude and phase are recombined into a complex spectrum, and then the frequency domain aligned features are obtained through inverse Fourier transform: ， Where, F FIM represents the features after frequency domain alignment, represents the real part operation, F -1 () represents inverse Fourier transform.

7. The method for full-color sharpening image fusion based on dual-domain flexible converter according to claim 6, wherein: The distribution is calibrated through the modulation mechanism, and the mean and variance of the MS are obtained through channel statistics. The modulation parameters are predicted by splicing the MS and PAN, and the PAN is reconstructed after instance normalization. The formula for reconstructing the PAN through instance normalization is: ， Where, F Mod Represents the distribution calibration characteristics, σ 2 represents the variance, μ represents the mean, γ 、 β represents the modulation parameters, IN () indicates instance normalization.

8. The method for full-color sharpening image fusion based on dual-domain flexible converter according to claim 7, wherein: The alignment feature is expressed as: ， Where, F AB Represents the alignment feature.

9. The method for full-color sharpening image fusion based on dual-domain flexible converter according to claim 8, wherein: In step S5, a stacked structure is first used to perform hierarchical fusion to F AB The image is input into the fusion network together with the MS, and the predicted image is output through the residual module, and then added to the MS image after bicubic interpolation to output the high-resolution multispectral image; wherein the calculation formula of the high-resolution multispectral image is: , Where, R () represents the residual module operation, represents the fusion network, MS bic Represents the upsampled low-resolution multispectral image.

10. A full-color sharpening image fusion device based on a dual-domain flexible converter, characterized in that: The device is applied to the full-color sharpening image fusion method based on the dual-domain flexible converter according to any one of claims 1 to 9; the device comprises: The feature extraction module is used to perform multi-layer space-frequency joint attention operations on the low-resolution multispectral image and the high-resolution panchromatic image to generate feature tensors MS and PAN; The feature alignment module is used to perform dual-domain feature alignment on MS and PAN, align structural semantic features based on the attention mechanism, fuse amplitude and phase spectrum information through Fourier transform, and use instance normalization to match modal distribution differences; The feature fusion module is used to fuse the alignment features and MS at multiple levels, performing concatenation, convolution, and spatial-frequency joint attention operations at each level to obtain fused features. A high-frequency reconstruction module is used to perform residual reconstruction on the fused features to generate a feature map with enhanced high-frequency details; The image generation module is used to add the reconstructed features to the upsampled low-resolution multispectral image element by element to output a high-resolution multispectral image.

Images