A multispectral image panchromatic sharpening method based on plug-and-play gradient feature-guided fusion

Abstract

This invention discloses a multispectral image panchromatic sharpening method based on plug-and-play gradient feature-guided fusion. Through the effective extraction and integration of gradient-guided features (GRAD), it achieves superior restoration results in terms of structure fidelity, detail preservation, and spectral consistency. Furthermore, the gradient feature-guided fusion is designed as a highly flexible plug-and-play method that can seamlessly enhance existing architectures. Specifically, this invention explicitly models GRAD by integrating the fine spatial details of high-resolution PAN with the intrinsic spectral information of MSI. Simultaneously, it employs an attention mechanism and learnable weighted and residual connections to achieve selective aggregation and adaptive fusion of image information, thereby significantly improving MSI quality. In addition, a systematic analysis of GRAD is performed from structural, detail, and spectral perspectives, and the results are optimized using a multi-objective loss function. This fully exploits gradient information, captures high-frequency details, achieves cross-modal structure-guided alignment, and flexibly and efficiently realizes multispectral image panchromatic sharpening, improving panchromatic sharpening performance.

Description

Technical Field

[0001] This invention belongs to the field of multispectral image pansharpening technology, and more specifically, relates to a multispectral image pansharpening method based on plug-and-play gradient feature-guided fusion. Background Technology

[0002] Multispectral images (MSI) play a crucial role in Earth science applications such as land cover classification, vegetation monitoring, and water body identification due to their rich spectral information. However, due to inherent limitations in satellite sensor performance and onboard processing capabilities, acquired MSI images typically have low spatial resolution, making it difficult to capture fine-grained information about complex surfaces. In contrast, panchromatic images (PAN) offer higher spatial resolution, capturing structural details more clearly, but their spectral information is relatively limited. To combine the advantages of both, researchers have conducted extensive studies to develop panchromatic sharpening techniques for multispectral remote sensing images. By fusing the high spatial resolution of panchromatic images with the rich spectral information of multispectral images, high-resolution multispectral images can be generated.

[0003] Current research on deep learning-based multispectral image panchromatic sharpening methods mainly falls into two categories. One category typically employs a dual-branch architecture to enhance the feature representation capabilities of specific modalities, focusing on the effective reconstruction of spectral and spatial information. These methods achieve decoupling and accurate restoration of spectral fidelity and spatial details by designing separate processing paths for panchromatic (PAN) and multispectral (MSI) images. The other category focuses on efficient fusion strategies for MSI and PAN, introducing attention mechanisms and Mamba structures to achieve cross-modal interaction and feature aggregation. Furthermore, convolution-based optimization techniques are often used as effective baseline solutions to improve structural prior representation capabilities and spatial alignment accuracy.

[0004] However, traditional multispectral image panchromatic sharpening methods often rely on fixed design patterns, making them difficult to adapt to different datasets or tasks. To address this issue, some recent studies have introduced plug-and-play (PnP) priors into the multispectral image panchromatic sharpening process, aiming to combine model-based regularization with learned prior information. Tao et al. proposed a PnP-based residual detail injection framework, and Shu et al. designed a dual-domain attention mechanism combining Transformer and CNN. Furthermore, some studies have applied PnP-based variational frameworks to jointly handle image misalignment and fusion problems, further validating the potential of PnP priors in improving the generalization and adaptability of multispectral image panchromatic sharpening tasks. Recently, a representative work is the Adaptive Two-Layer Weighted Mechanism (ADWM) proposed by Huang et al., which utilizes the covariance matrix to model the redundancy and heterogeneity between and within feature layers, thereby reducing information overlap and enhancing feature discriminative power. Although ADWM is not explicitly designed as a PnP block, it functions similarly, serving as an effective enhancement component integrated into the sharpening process.

[0005] Existing PnP-based multispectral image panchromatic sharpening methods still have limited utilization of structure priors, especially in terms of gradient-based structure guidance, which has not been fully explored. Furthermore, few current multispectral image panchromatic sharpening methods offer a modular and flexible framework to seamlessly integrate gradient cues across different panchromatic sharpening architectures.

[0006] Existing multispectral image panchromatic sharpening methods have the following drawbacks: 1. They rely on fixed design patterns, making them difficult to adapt to different datasets or tasks; 2. The utilization of structural priors is still relatively limited, especially the structural guidance based on gradient information has not been fully explored; 3. There is currently a lack of a flexible and modular framework that can seamlessly integrate gradient priors into existing architectures, resulting in shortcomings in capturing high-frequency details and achieving cross-modal structural guidance alignment. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multispectral image panchromatic sharpening method based on plug-and-play gradient feature-guided fusion, so as to fully mine gradient information, capture high-frequency details, and achieve cross-modal structure-guided alignment, thereby flexibly and efficiently realizing multispectral image panchromatic sharpening and improving panchromatic sharpening performance.

[0008] To achieve the above-mentioned objectives, this invention provides a multispectral image panchromatic sharpening method based on plug-and-play gradient feature-guided fusion, characterized by the following steps:

[0009] (1) Gradient feature extraction

[0010] Based on low-resolution multispectral images MSI Panchromatic Image I PAN Extract gradient-guided features;

[0011] (2) Gradient-guided fusion

[0012] 2.1) Normalization and convolution mapping are used to obtain the query, key, and value.

[0013] The gradient-guided feature GRAD is normalized, and then passed through a GRAD convolution mapping to obtain the value V. GRAD Key K GRAD For panchromatic image I PAN After normalization, the values ​​are then passed through a PAN convolution mapping to obtain the value V. PAN Key K PAN For multispectral images After normalization, and then through an MSI convolution mapping, the value V is obtained. MSI Key K MSI And query Q MSI ;

[0014] 2.2) Calculate attention

[0015] Value V GRAD Key K GRAD And query Q MSI Attention is obtained after passing through the attention module. GRAD Value V PAN Key K PAN And query Q MSI Attention is obtained after passing through the attention module. PAN Value V MSI Key K MSI And query Q MSI Attention is obtained after passing through the attention module. MSI ;

[0016] 2.3) Generate MSI features

[0017] Attention GRAD Attention PA Attention MSI The MSI features F are modulated using learnable weights α, β, and γ, respectively, and then aggregated through weighted summation and refined by a convolutional layer. MSI :

[0018] F MSI =Conv(Attention) GRAD⊙α+Attention PAN ⊙β+Attention MSI ⊙γ)

[0019] (3) Generate high-resolution multispectral images

[0020] MSI feature F MSI Input to the baseline model, at the insertion point, and multispectral image Panchromatic Image I PAN Gradient-guided fusion is performed, and the fused output is mapped to the corresponding target dimension through a convolutional layer to obtain a high-resolution multispectral image.

[0021] (4) Joint optimization of losses

[0022] Calculate the total loss function

[0023] (5) Training a multispectral image panchromatic sharpening system

[0024] Based on steps (1), (2), and (3), a multispectral image panchromatic sharpening system is constructed, and based on the total loss function... The network parameters of the multispectral image panchromatic sharpening system are updated using the gradient descent method until the set conditions are met.

[0025] (6) Multispectral image full-color sharpening

[0026] Multispectral image I MSI Panchromatic Image I PAN The image is fed into a multispectral image panchromatic sharpening system and processed according to steps (1), (2), and (3) to obtain a high-resolution multispectral image.

[0027] The objective of this invention is achieved as follows.

[0028] This invention presents a plug-and-play gradient feature-guided fusion-based multispectral image panchromatic sharpening method. Through the effective extraction and integration of gradient-guided features (GRAD), it achieves superior restoration results in terms of structure fidelity, detail preservation, and spectral consistency. This invention designs gradient feature-guided fusion as a highly flexible plug-and-play method that can seamlessly enhance existing architectures. Specifically, this invention explicitly models GRAD by integrating the fine spatial details of high-resolution PANs with the intrinsic spectral information of MSIs. Simultaneously, it employs attention mechanisms and learnable weighted and residual connections to achieve selective aggregation and adaptive fusion of image information, thereby significantly improving MSI quality. Furthermore, it systematically analyzes GRADs from structural, detail, and spectral perspectives, optimizing the results through a multi-objective loss function. This fully leverages gradient information, captures high-frequency details, achieves cross-modal structure-guided alignment, and flexibly and efficiently realizes multispectral image panchromatic sharpening, improving panchromatic sharpening performance. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the structure of the multispectral image panchromatic sharpening system designed under the plug-and-play gradient feature-guided fusion method of the present invention;

[0030] Figure 2 This is a flowchart of a specific implementation of the multispectral image panchromatic sharpening method based on plug-and-play gradient feature-guided fusion according to the present invention;

[0031] Figure 3 This is a schematic diagram of the gradient-guided fusion principle;

[0032] Figure 4 This is a comparison chart of the average channel differences between the high-resolution multispectral image obtained by full-color sharpening in this invention and the reference image. Detailed Implementation

[0033] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0034] Figure 1 This is a schematic diagram of the structure of the multispectral image panchromatic sharpening system designed under the plug-and-play gradient feature-guided fusion method of the present invention.

[0035] To address the shortcomings of existing panchromatic sharpening methods in capturing high-frequency details and achieving cross-modal structure-guided alignment, this invention constructs a multispectral image panchromatic sharpening system, namely a flexible and efficient plug-and-play module, such as... Figure 1As shown, this invention revolves around three key issues: how to effectively extract gradient information (GRAD), how to incorporate it into the fusion process, and how to optimize the fusion method to improve reconstruction performance. Correspondingly, there are a gradient feature extraction module, a gradient-guided fusion module, and a loss joint optimization module (connected by dashed lines). The processing procedure of each module is as follows: Figure 2 As shown.

[0036] Figure 2 This is a flowchart of a specific implementation of the multispectral image panchromatic sharpening method based on plug-and-play gradient feature-guided fusion according to the present invention.

[0037] In this embodiment, as Figure 1 , 2 As shown, the multispectral image panchromatic sharpening method based on plug-and-play gradient feature-guided fusion of the present invention includes the following steps:

[0038] Step S1: Gradient Feature Extraction

[0039] Step S1.1: Shallow gradient feature extraction

[0040] To explicitly model the fused gradient features, shallow extraction of the original gradient features of MSI and PAN is performed, and the panchromatic image PAN(I) is explicitly modeled. PAN ) and multispectral image MSI (I MSI The structural priors included in the model first extract shallow gradient features from these two modalities. These initial features highlight local edge and texture information, providing a foundation for subsequent deeper abstract learning. Specifically:

[0041] For low-resolution multispectral images I MSI Upsampling is performed to obtain the image I (similar to the panchromatic image). PAN Multispectral images of the same size Then, shallow gradient features are extracted:

[0042]

[0043] in, G represents a gradient operator, such as the Sobel operator. MSI For shallow gradient features of multispectral images, G PAN This represents the shallow gradient features of a panchromatic image.

[0044] Step S1.2: Deep gradient feature extraction

[0045] To capture more semantically meaningful and discriminative high-frequency representations, we further... MSI and G PAN Input their respective encoders f MSI (·) and f PANProcessing in (·):

[0046] Φ MSI =f MSI (G MSI ),Φ PAN =f PAN (G PAN )

[0047] Among them, f MSI (·) represents the encoder of the multispectral image, f PAN (·) represents the encoder of the panchromatic image, Φ MSI For deep gradient features of multispectral images, Φ PAN These are deep gradient features of panchromatic images. These deep gradient features represent higher-level gradient semantic information, such as complex texture primitives or robust boundary structures, and are less sensitive to noise and offset.

[0048] Step S1.3: Gradient-guided feature extraction

[0049] Obtaining deep gradient features Φ MSI and Φ PAN Then, they are first spliced ​​together to form a joint representation, and then fused through a network. To model the feature interaction relationship between the two, a channel attention module is then introduced. To further enhance the fusion result, this module adaptively reweights the features of each channel based on their importance. The final enhanced gradient features are obtained through residual connections. This residual enhancement mechanism ensures that the fused gradient features are structurally consistent with the PAN image and also spectrally consistent with the MSI image. Specifically:

[0050]

[0051] Here, GRAD stands for Gradient Guided Features, and concat indicates concatenation by channel. Indicates a converged network. This indicates the channel attention module.

[0052] Step S2: Gradient-guided fusion

[0053] To better apply the gradient feature information from the fused representation to the entire pansharpening process, this invention designs an attention module aimed at injecting spatial details into the MSI from PAN and GRAD features, and enhancing it using the spatial context of the MSI itself, thereby achieving selective aggregation and adaptive fusion of spatial information. This part mainly includes three pathways, such as... Figure 3 As shown. Specifically, gradient-guided fusion includes the following steps:

[0054] Step S2.1: Normalization and convolution mapping to obtain the query, key, and value.

[0055] The gradient-guided feature GRAD is normalized, and then passed through a GRAD convolution mapping to obtain the value V. GRAD Key K GRAD For panchromatic image I PAN After normalization, the values ​​are then passed through a PAN convolution mapping to obtain the value V. PAN Key K PAN For multispectral images I MSI After normalization, and then through an MSI convolution mapping, the value V is obtained. MSI Key K MSI And query Q MSI .

[0056] Step S2.2: Calculate attention

[0057] Value V GRAD Key K GRAD And query Q MSI Attention is obtained after passing through the attention module. GRAD Value V PAN Key K PAN And query Q MSI Attention is obtained after passing through the attention module. PAN Value V MSI Key K MSI And query Q MSI Attention is obtained after passing through the attention module. MSI .

[0058] In this invention, GRAD-guided texture refinement selectively injects fine-grained edge and texture information from GRAD into the MSI representation to enrich its high-frequency details; PAN-guided spatial detail injection adaptively selects and weights spatial detail information for each spatial location in the MSI feature map according to the corresponding row in the PAN feature map, thereby enhancing spatial fidelity; and spectral-contextual association models inter-channel dependencies and long-distance spatial context information within the MSI, thereby achieving better structure-aware enhancement and spectral consistency.

[0059] Step S2.3: Generate MSI features

[0060] Attention GRAD Attention PA Attention MSIThe MSI features F are modulated using learnable weights α, β, and γ, respectively, and then aggregated through weighted summation and refined by a convolutional layer. MSI :

[0061] F MSI =Conv(Attention) GRAD ⊙α+Attention PAN ⊙β+Attention MSI ⊙γ)

[0062] In this invention, the gradient-guided fusion module aims to generate high-quality MSI features that possess both fine spatial detail and maintain spectral accuracy by jointly modeling the spatial information of the PAN, the texture details of the GRAD, and the spectral contextual cues of the MSI. Simultaneously, it effectively reduces spectral distortion and spatial artifacts caused by the uneven correlation between the PAN and the MSI.

[0063] Step S3: Generate high-resolution multispectral images

[0064] MSI feature F MSI Input to the baseline model, at the insertion point, and multispectral image Panchromatic Image I PAN Gradient-guided fusion is performed, and the fused output is mapped to the corresponding target dimension through a convolutional layer to obtain a high-resolution multispectral image.

[0065] Step S4: Joint Loss Optimization

[0066] Step S4.1: Calculate pixel-level reconstruction loss

[0067] Pixel-level reconstruction loss is based on the L1 norm, a widely used loss mechanism in image restoration tasks. Its goal is to minimize the absolute difference between the predicted image and the high-resolution ground truth image, thereby promoting structural similarity and maintaining consistency in the global intensity distribution. Specifically:

[0068]

[0069] in, It is a high-resolution multispectral image. The value at pixel x, It is a high-resolution true multispectral image. The value at pixel x, Ω represents the set of all pixel positions, and N is the total number of pixels.

[0070] Step S4.2: Calculate gradient detail loss

[0071] Pixel-level reconstruction loss based on the L1 norm is typically insensitive to high-frequency components, often leading to overly smooth image edges and loss of fine-grained texture details. As a representative of high-frequency information, image gradients can effectively capture local texture, contour, and structural variations. Introducing gradient detail loss can compensate for the shortcomings of the L1 norm by explicitly enhancing edge and texture reconstruction, thereby improving the perceptual sharpness and structural awareness of the output image, especially in spatially rich regions. Specifically:

[0072]

[0073] in, This indicates the calculation of the image gradient.

[0074] Step S4.3: Calculate the spectral uniformity loss

[0075] For panchromatic sharpening tasks, maintaining consistency across spectral channels while preserving spatial details is equally crucial. Spectral consistency loss is used to constrain the channel relationships between the reconstructed image and the reference image, thereby reducing spectral distortions such as color shifts or band misalignments, and improving the realism and practicality of the reconstructed image in remote sensing applications. Specifically:

[0076]

[0077] Step S4.4: Calculate the total loss function

[0078] The three loss terms above are functionally complementary: pixel-level reconstruction loss enhances structure alignment, gradient detail loss improves detail sharpness, and spectral consistency loss maintains consistency across bands. Through joint optimization, the total loss function... for:

[0079]

[0080] Step S5: Training the multispectral image panchromatic sharpening system

[0081] A multispectral image panchromatic sharpening system is constructed based on steps S1, S2, and S3, and the total loss function is used as the basis for this system. The network parameters of the multispectral image panchromatic sharpening system are updated using the gradient descent method until the set conditions are met.

[0082] Step S6: Multispectral Image Pancolor Sharpening

[0083] Multispectral image I MSI Panchromatic Image I PAN The image is fed into a multispectral image panchromatic sharpening system and processed according to steps S1, S2, and S3 to obtain a high-resolution multispectral image.

[0084] Example

[0085] This invention utilizes a plug-and-play gradient feature-guided fusion method for multispectral image panchromatic sharpening. The panchromatic sharpening dataset proposed by Deng et al. was trained and tested on multiple baseline models (PanNet, FusionNet). This dataset includes WV3, QB, and GF2 satellite data. The results were evaluated using PSNR, SSIM, ERGAS, SAM, SCC, and Q2n metrics. The comparison results are shown in Table 1.

[0086]

[0087] Table 1

[0088] Table 1 shows the improvement results of the present invention on multiple baselines in the WV3, QB, and GF2 datasets. The method proposed in this invention is highlighted in bold, showing a significant improvement compared to the baseline models.

[0089] The average channel difference between the generated high-resolution multispectral image and the high-resolution real multispectral image (reference image) is visualized as follows: Figure 4 As shown, the left column of each group displays the baseline results, while the right column displays the results enhanced by the method of this invention; brighter colors indicate greater differences. The method proposed in this invention effectively reduces edge artifacts, improves the fidelity of structural details, and verifies its effectiveness in high-frequency reconstruction.

[0090] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A multi-spectral image panchromatic sharpening method based on plug-and-play gradient feature guided fusion, characterized in that, Includes the following steps: (1) Gradient feature extraction Based on low-resolution multispectral images I MSI , panchromatic image I PAN Extract gradient-guided features GRAD; (2) Gradient-guided fusion 2.1) Normalization and convolution mapping are used to obtain the query, key, and value. The gradient-guided feature GRAD is normalized, and then passed through a GRAD convolution mapping to obtain the value V. GRAD Key K GRAD For panchromatic image I PAN After normalization, the values ​​are then passed through a PAN convolution mapping to obtain the value V. PAN Key K PAN For multispectral images After normalization, and then through an MSI convolution mapping, the value V is obtained. MSI Key K MSI And query Q MSI ; 2.2) Calculate attention value V GRAD , key K GRAD and query Q MSI get attention Attention GRAD , value V PAN , key K PAN and query Q MSI get attention Attention PAN , value V MSI , key K MsI and query Q MSI get attention Attention MSI ; 2.3) Generate MSI features Attention GRAD , Attention PAN , Attention MSI are respectively modulated by learnable weights a, b, g, and finally aggregated by weighted summation, and refined by a convolution layer to obtain the MSI feature F MSI : F MSI = Conv(Attention GRAD ⊙ α + Attention PAN ⊙ β + Attention MSI ⊙ γ) (3) Generate high-resolution multispectral images MSI feature F MSI Input to the baseline model, at the insertion point, and multispectral image Panchromatic Image I PAN Gradient-guided fusion is performed, and the fused output is mapped to the corresponding target dimension through a convolutional layer to obtain a high-resolution multispectral image. (4) Joint Loss Optimization Calculate the total loss function (5) Training a multispectral image panchromatic sharpening system Based on steps (1), (2), and (3), a multispectral image panchromatic sharpening system is constructed, and based on the total loss function... The network parameters of the multispectral image panchromatic sharpening system are updated using the gradient descent method until the set conditions are met. (6) Multispectral image full-color sharpening Multispectral image I MSI Panchromatic Image I PAN The image is fed into a multispectral image panchromatic sharpening system and processed according to steps (1), (2), and (3) to obtain a high-resolution multispectral image.

2. The multispectral image panchromatic sharpening method based on plug-and-play gradient feature-guided fusion according to claim 1, characterized in that, The gradient feature extraction described in step (1) is as follows: 1.1) Shallow gradient feature extraction For low-resolution multispectral images I MSI Upsampling is performed to obtain the image I (similar to the panchromatic image). PAN Multispectral images of the same size Then, shallow gradient features are extracted: in, Let G represent the gradient operator. MSI For shallow gradient features of multispectral images, G PAN This represents the shallow gradient features of a panchromatic image. 1.2) Deep gradient feature extraction Φ MSI = f MSI (G MSI ), Φ PAN = f PAN (G PAN ) where f MSI (·) denotes the encoder of the multispectral image, f PAN (·) denotes the encoder of the panchromatic image, Φ MSI is a deep gradient feature of the multispectral image, Φ PAN is a deep gradient feature of the panchromatic image; 1.3) Gradient-guided feature extraction Here, GRAD stands for Gradient Guided Features, and concat indicates concatenation by channel. Indicates a converged network. This indicates the channel attention module.

3. The multispectral image panchromatic sharpening method based on plug-and-play gradient feature-guided fusion according to claim 1, characterized in that, The joint optimization of the loss described in step (4) is as follows: 4.1) Calculate pixel-level reconstruction loss in, It is a high-resolution multispectral image. The value at pixel x, It is a high-resolution true multispectral image. The value at pixel x, Ω represents the set of all pixel positions, and N is the total number of pixels; 4.2) Calculate gradient detail loss in, This indicates the calculation of image gradient; 4.3) Calculate the spectral uniformity loss 4.4) Calculate the total loss function

Images