Multispectral panchromatic sharpening method based on wavelet transform diffusion model
By processing low-frequency and high-frequency components through wavelet transform and cross-modal attention mechanism, the problems of high computational cost and insufficient modal fusion in panchromatic sharpening of multispectral images are solved, and efficient and high-quality image generation is achieved.
Patent Information
- Application Number
- CN202610761202.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-29
- Publication Date
- 2026-08-25
AI Technical Summary
Existing deep learning methods cannot take into account both local and global features in panchromatic sharpening of multispectral images, have high computational costs, are unstable in the training process of generative models, and cannot effectively utilize the complementary properties between modalities.
Wavelet transform is used to decompose the image into low-frequency and high-frequency components. The low-frequency components are processed by conditional guidance, and the high-frequency components are processed by cross-modal attention mechanism. Feature fusion is performed by combining a multi-scale purification module, and finally the image is reconstructed by inverse transform.
It significantly reduces computational load, improves computational efficiency and generation quality, achieves deep coupling of spectral and spatial information, and generates images with excellent spectral fidelity and spatial resolution, overcoming the problems of insufficient modal fusion and loss of detail in existing technologies.
Smart Images
Figure CN122636451A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of remote sensing image processing technology, and relates to a multispectral panchromatic sharpening method based on a wavelet transform diffusion model. Background Technology
[0002] Advances in satellite remote sensing technology have increased the demand for high-resolution multispectral (HRMS) imagery across various fields. However, due to the hardware limitations of existing physical sensors, HRMS imagery cannot be acquired directly. Therefore, pansharpening technology, by fusing high spatial resolution panchromatic (PAN) imagery with high spectral resolution low-resolution multispectral (LRMS) imagery, has become a primary method for acquiring HRMS imagery.
[0003] Traditional panchromatic sharpening methods mainly fall into three categories: Component Substitution (CS) methods are simple to implement but prone to significant spectral distortion; Multi-Resolution Analysis (MRA) methods have high spectral fidelity but often introduce spatial artifacts; Variational Optimization (VO) methods have better overall performance but high computational complexity.
[0004] Deep learning methods have shown great potential in this field, but they still face their own technical bottlenecks. CNNs extract local features through multiple layers of convolution, resulting in strong local details, but they struggle to effectively capture long-range dependencies and cannot model global context. Transformers have strong global modeling capabilities and support cross-modal fusion, but they are computationally expensive and prone to losing local details. Diffusion models can generate high-quality details with excellent fidelity, but the numerous sampling steps lead to slow inference speed, poor stability during training, and limited precision in controlling panchromatic and low-resolution multispectral conditional information, making it difficult to efficiently utilize the complementary characteristics between modalities.
[0005] Existing CNNs and Transformers cannot take into account both local and global features, and have high computational costs, making generative models unable to effectively guide image generation. Summary of the Invention
[0006] In view of this, the purpose of this invention is to provide a multispectral panchromatic sharpening method based on a wavelet transform diffusion model. The method transforms the image to the frequency domain using wavelet transform and processes low and high frequencies using different methods to reduce computational load. For the low-frequency part, the model is guided to generate the image through conditional guidance to improve the image generation quality. For the high-frequency part, an attention mechanism is used with LRMS as Q and pan as k and v to perform cross-modal interaction to enhance high-frequency information. Finally, the complete HRMS image is obtained through inverse transform.
[0007] To achieve the above objectives, the present invention provides the following technical solution: A multispectral panchromatic sharpening method based on a wavelet transform diffusion model, the method comprising: S1. Acquire remote sensing satellite image datasets, including ground truth images (GT), panchromatic images (PAN), and low-resolution multispectral images (LRMS), and perform preprocessing. S2. Decompose the panchromatic image PAN, the low-resolution multispectral image LRMS, and the real image GT by discrete wavelet transform, and extract their low-frequency and high-frequency components respectively. S3. Construct a low-frequency processing branch, in which the panchromatic image PAN and the low-resolution multispectral image LRMS are first input into the feature interaction module to extract fusion condition features. Then, a forward noise addition process is constructed by adding noise to the low-frequency components of the real image GT and the low-resolution multispectral image LRMS, and a reverse noise reduction process is constructed guided by the fusion condition features output by the feature interaction module to obtain the final output low-frequency features. S4. Construct a high-frequency processing branch, in which the high-frequency features of the low-resolution multispectral image LRMS are used as query Q, and the high-frequency components of the panchromatic image PAN are used as key K and value V to perform cross-modal attention calculation. Multi-scale feature extraction is performed through a multi-scale purification module for adaptive fusion to obtain the final output high-frequency features. S5. Perform inverse discrete wavelet transform (IWT) on the obtained low-frequency and high-frequency features to reconstruct the final multispectral panchromatic sharpened image.
[0008] Furthermore, in step S1, real images GT, panchromatic images PAN, and low-resolution multispectral images LRMS are obtained from publicly available remote sensing data resources. All image data are preprocessed using data normalization to unify the range of image pixel values and complete data standardization.
[0009] Furthermore, in step S2, for any two-dimensional image signal After discrete wavelet transform decomposition, one low-frequency approximate component and three high-frequency detail components are obtained, which are represented as follows:
[0010] in, It represents low-frequency components, preserving the overall outline and brightness information of the image; The high-frequency components in the horizontal direction correspond to the vertical edge details; The high-frequency components in the vertical direction correspond to the horizontal edge details; These are the high-frequency components in the diagonal direction, corresponding to texture details; For the input real image GT, panchromatic image PAN, and low-resolution multispectral image LRMS, the decomposed low-frequency and high-frequency components are represented as follows: , , .
[0011] Furthermore, in step S3, during the forward process, the low-frequency components of the real image GT are... Low-frequency components of low-resolution multispectral images LRMS Noise is gradually added to the residual image. In the forward process, Gaussian noise is gradually added to the clean residual between the real image and the low-frequency component of the multispectral image according to a preset ratio until it becomes pure noise. This is used as training data for the neural network to learn inverse denoising. In the reverse process, guided by the conditional fusion features output by the feature interaction module, a high-quality fusion result is gradually restored in the low-frequency domain. Its conditional denoising process can be represented as follows:
[0012] Among them, condition information That is, the output of the feature interaction module and The low-frequency components guide the model to recover the low-frequency details and texture information of the high-resolution multispectral image HRMS. The conditional denoising process is a parameterized Markov chain, where the probability represents the neural network's probability estimate of the inverse denoising trajectory. The final high-quality fusion result is the residual of the low-frequency components of the high-resolution multispectral image HRMS. The final image is obtained by superimposing the low-frequency signal of LRMS. .
[0013] Furthermore, the feature interaction module in step S3 includes a channel interaction module and a spatial-level interaction module, wherein, Channel interaction module receives and As input, global average pooling and max pooling operations are performed on both respectively, compressing the spatial dimension into channel descriptors, which are then concatenated to obtain... global vector :
[0014] Then, channel attention weights for LRMS and PAN are generated separately via MLP. and The two sets of weights are multiplied element-wise with the original input and then summed to achieve a weighted sum of cross-modal channel-level features, outputting... and :
[0015]
[0016]
[0017] The spatial-level interaction module receives the output of the first stage. and splicing the two together The combined features, while through average pooling, Convolution, max pooling, and other three paths extract spatial statistics. :
[0018] Spatial attention masks are then generated using MLP. and The spatial masks are applied to their respective input features, and cross-modal spatial feature selection and enhancement are achieved through element-wise multiplication and residual addition, ultimately outputting high-quality features. and :
[0019]
[0020]
[0021] in, This represents the Sigmoid activation function. This indicates element-wise multiplication.
[0022] Furthermore, in step S3, the training objective of the conditional diffusion model adopts a noise prediction method. Parameter optimization is achieved by minimizing the mean square error between the actual noise and the predicted noise. Its loss function is defined as follows:
[0023] in, It is real Gaussian noise. This loss function is used to predict noise for the model. By continuously optimizing this loss function, the model can gradually learn the data distribution and the relationships between constraints.
[0024] Furthermore, in step S4, the high-frequency processing branch takes the high-frequency sub-bands of the PAN and LRMS images after discrete wavelet transform decomposition as input, and achieves deep fusion of high-frequency features through attention guidance and multi-scale refinement. The process includes: For the high-frequency subbands of PAN, K and V are generated through normalization and convolution operations, respectively; for the high-frequency subbands of LRMS, Q is generated using the same method.
[0025] Q and K are normalized using SoftMax and then multiplied element-wise to obtain the attention weight map S. This S is then multiplied by V to extract cross-modal attention features using LRMS as the query and PAN as the key. Subsequently, the attention output is convolved and then added to the residual of the original LRMS high-frequency features to obtain the cross-attention features.
[0026] in, Represents the convolution weights. For learnable residual scaling factors, The scaling factor is used to calculate the high-frequency components of the input panchromatic image PAN. and low-resolution multispectral images The extracted cross-modal attention features are respectively representing ; The obtained features are input into two cascaded multi-scale refinement modules for multi-scale refinement. The refinement results are then fused with another feature output through PAN attention by Conv1×1 convolution, activation function PReLU, and Conv3×3 convolution. Finally, the original LRMS high-frequency residuals are superimposed through convolution and addition operations to output the enhanced HH, HL, and LH high-frequency subbands.
[0027] Furthermore, the input features of the multi-scale purification module in step S4 are simultaneously fed into three parallel branches: the first branch uses standard Conv3×3 convolution to capture local texture information; the second branch also uses Conv3×3 to extract medium-scale features; the third branch uses Conv3×3 dilated convolution to expand the receptive field and capture a wider range of contextual information; the outputs of the three branches are all concatenated after PReLU activation to form a multi-scale joint representation; the concatenated features are sequentially processed by Conv3×3, PreLU, and Conv1×1 for channel compression and nonlinear transformation, and then fed into a channel attention branch: important weights for each channel are generated through pooling and activation functions, and multiplied element-wise with the main features to achieve adaptive selection of features at different scales; finally, the weighted features are added to the original input residuals to output the enhanced feature map; the input features The process is represented as follows:
[0028]
[0029] In the formula, For feature fusion and channel dimensionality reduction operations, For channel attention mapping function; Finally, the high-frequency features in the LH, HL, and HH directions are concatenated, and the global correction tensor Call is predicted through a nonlinear convolutional network. This tensor is then split into three independent tensors and weighted to their respective frequency bands.
[0030]
[0031] In the formula, This represents the high-frequency correction weighting coefficient, used to control the correction strength of the global correction tensor for the high-frequency characteristics of each frequency band. Before cross-band interaction, the first Initial fusion high-frequency characteristics of each frequency band.
[0032] Furthermore, in step S5, the final output image from step S3 is... The final output image of step S4 Performing the inverse discrete wavelet transform (IWT) yields the final output result, and the process is represented as follows:
[0033] In the formula, It is the inverse discrete wavelet transform function.
[0034] Furthermore, the training process for this method consists of three training phases, including: Phase 1 Low-frequency pre-training: Fix the high-frequency processing branch and train only the low-frequency processing branch; Phase 2 High-Frequency Pre-training: Fix the low-frequency processing branch and train only the high-frequency processing branch; Phase 3 end-to-end joint fine-tuning: Unfreeze all branches, jointly optimize low-frequency and high-frequency branches; reconstruct the complete HRMS image by inverse transformation of the predicted low-frequency components and predicted high-frequency subbands; The combined loss function, which includes low-frequency diffusion loss, high-frequency structure loss, and full-image reconstruction loss, is calculated. The model parameters are updated through backpropagation, and the weights of each loss are dynamically adjusted in stages.
[0035] The beneficial effects of this invention are as follows: First, this invention significantly reduces the overall computational load and improves computational efficiency by transforming the image to the frequency domain and performing divide-and-conquer processing on the low-frequency and high-frequency components. Compared to traditional methods that directly extract and generate global features from the entire image in the original high-resolution spatial domain, this invention first uses Discrete Wavelet Transform (DWT) to decompose the input multispectral (LRMS) and panchromatic (PAN) images into low-frequency approximate components and high-frequency detail components. The low-frequency components, with their spatial size halved, have only one-quarter the data volume of the original image. Subsequent conditional guidance and generation processes on these components directly reduce the computational complexity of this stage by more than an order of magnitude. Meanwhile, the high-frequency components mainly carry edge and texture information. This invention uses a lightweight attention mechanism and residual connections to process them, avoiding redundant global computations in the high-frequency detail domain. Through this frequency domain decomposition and task splitting approach, this invention effectively avoids the inherent huge computational overhead of high-resolution image generation in traditional methods, significantly improving model training and inference speeds, significantly reducing computational resource consumption, and achieving efficient image fusion.
[0036] Secondly, this invention effectively reduces feature differences between different modes by introducing cross-modal information interaction mechanisms in both the low-frequency and high-frequency stages, resulting in more complete fusion of multispectral and panchromatic images and significantly improving the overall quality of the fused image. In the low-frequency component processing stage, this invention uses a Feature Interaction Module (MIM) to perform deep feature interaction on the low-frequency components of the LRMS and PAN images, learning and weakening the feature distribution differences between the two modes in the low-frequency domain. The processed low-frequency features are then used as conditions to guide the generation process of the low-frequency components. This mechanism allows the generated low-frequency components to simultaneously retain the spectral characteristics of the LRMS image and the overall contour information of the PAN image, avoiding spectral distortion or spatial structure shift problems that are prone to occur under single-modal guidance. In the high-frequency component processing stage, this invention achieves effective interaction and complementarity of high-frequency detail features between the two modes through an attention mechanism, and combines this with residual connections for feature fusion. This allows the fused high-frequency components to accurately inherit the high-resolution spatial details of the PAN image, while avoiding information redundancy and noise amplification problems caused by direct stitching or simple weighting. By employing a two-stage cross-modal interaction between low and high frequencies, this invention systematically reduces the feature gap between different modes, achieving deep coupling of spectral information and spatial details. The resulting fused image exhibits excellent performance in both spectral fidelity and spatial resolution, effectively overcoming the defects commonly found in existing fusion technologies, such as insufficient modal fusion, loss of details, or spectral distortion.
[0037] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0038] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall architecture of the multispectral panchromatic sharpening method based on the wavelet transform diffusion model in an embodiment of the present invention; Figure 2 This is a schematic diagram of the feature interaction module in an embodiment of the present invention; Figure 3 This is a schematic diagram of the low-frequency processing branch in an embodiment of the present invention; Figure 4 This is a schematic diagram of the high-frequency processing branch in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the multi-scale purification module according to an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the visualization comparison results of the generated HRMS images under an embodiment of the present invention, wherein, Figure 6 (a) is the original image. Figure 6 (j) is the HRMS image of the method of the present invention. Figures (b)-(i) are the HRMS images of the existing GS method, CNMF method, IHS method, PNN method, MSDCNN method, difpan method, crossdif method, and pandit method, respectively. Detailed Implementation
[0039] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0040] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0041] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.
[0042] Please see Figures 1-6 This is a multispectral panchromatic sharpening method based on a wavelet transform diffusion model.
[0043] Example 1 This embodiment first describes the specific implementation process of the multispectral panchromatic sharpening method based on the wavelet transform diffusion model. Figure 1 The overall network architecture of this method is illustrated in the diagram, which specifically includes the following steps: S1. Acquire remote sensing satellite image datasets, including ground truth images (GT), panchromatic images (PAN), and low-resolution multispectral images (LRMS), and perform preprocessing. S2. Decompose the panchromatic image PAN, the low-resolution multispectral image LRMS, and the real image GT by discrete wavelet transform, and extract their low-frequency and high-frequency components respectively. S3. Construct a low-frequency processing branch, in which the panchromatic image PAN and the low-resolution multispectral image LRMS are first input into the feature interaction module to extract fusion condition features. Then, a forward noise addition process is constructed by adding noise to the low-frequency components of the real image GT and the low-resolution multispectral image LRMS, and a reverse noise reduction process is constructed guided by the fusion condition features output by the feature interaction module to obtain the final output low-frequency features. S4. Construct a high-frequency processing branch, in which the high-frequency features of the low-resolution multispectral image LRMS are used as query Q, and the high-frequency components of the panchromatic image PAN are used as key K and value V to perform cross-modal attention calculation. Multi-scale feature extraction is performed through a multi-scale purification module for adaptive fusion to obtain the final output high-frequency features. S5. Perform inverse discrete wavelet transform (IWT) on the obtained low-frequency and high-frequency features to reconstruct the final multispectral panchromatic sharpened image.
[0044] In step S1 of this embodiment, a remote sensing satellite image dataset is acquired. This dataset consists of ground truth (GT) images, panchromatic (PAN) images, and low-resolution multispectral (LRMS) images. The GT images, PAN images, and LRMS images are all obtained from publicly available remote sensing data resources. After image acquisition, all image data is preprocessed. This preprocessing step employs data normalization to standardize the range of image pixel values.
[0045] In step S2 of this embodiment, for any two-dimensional image signal After a first-order discrete wavelet transform (DWT), it can be decomposed into one low-frequency approximation component (LL) and three high-frequency detail components (LH, HL, HH), denoted as:
[0046] in, It represents low-frequency components, preserving the overall outline and brightness information of the image; The high-frequency components in the horizontal direction correspond to the vertical edge details; The high-frequency components in the vertical direction correspond to the horizontal edge details; The high-frequency components in the diagonal direction correspond to texture details.
[0047] In this embodiment, the low-frequency and high-frequency components of the obtained real image GT are respectively represented as: The low-frequency and high-frequency components of the obtained panchromatic image PAN are represented as follows: The low-frequency and high-frequency components of the obtained low-resolution multispectral image LRMS are represented as follows: .
[0048] In step S3 of this embodiment, the MIM feature interaction module consists of two cascaded sub-modules, designed to achieve bidirectional feature interaction and mutual enhancement between LRMS low-frequency and PAN low-frequency signals. The two sub-modules are a channel interaction module and a spatial-level interaction module, respectively. Figure 2 As shown, the channel interaction module receives and As input, global average pooling and max pooling operations are performed on both respectively, compressing the spatial dimension into channel descriptors, which are then concatenated to obtain... global vector Then, channel attention weights for LRMS and PAN are generated separately via MLP. and These two sets of weights are multiplied element-wise with the original input and then summed to achieve a weighted sum of cross-modal channel-level features, outputting... and The space-level interaction module receives the output from the first stage. and splicing the two together The combined features, while through average pooling, Spatial statistics are extracted through three paths: convolution, max pooling, and MLP, which then generate spatial attention masks. and These two spatial masks are applied to their respective input features. Through element-wise multiplication and residual addition, cross-modal spatial feature selection and enhancement are achieved, ultimately outputting high-quality features. and The specific process is as follows: First, initial feature extraction is performed on the input PAN and LRMS images, mapping them to the same channel dimension:
[0049] During the spectral information calibration phase, the model extracts channel features along the spatial dimension using global average pooling (GAP) and global maximum pooling (GMP). :
[0050] Subsequently, a multilayer perceptron (MLP) is used to generate bimodal spectral calibration weights, and the residuals are weighted to the initial features:
[0051]
[0052]
[0053] in, This represents the Sigmoid activation function. This indicates element-wise multiplication.
[0054] In the subsequent spatial information calibration stage, we concatenate the above features along the channel dimension, calculate their average and maximum values, and combine them with the cross-channel local responses extracted by 1x1 convolution to construct spatial features. :
[0055] Similarly, spatial calibration weights are generated using MLP to obtain the final interaction condition features:
[0056]
[0057]
[0058] Overall, the MIM module, through a two-stage bidirectional attention mechanism of channel and space, enables the features of LRMS and PAN to be mutually conditional and mutually guiding, effectively integrating spectral information and spatial details, and providing high-quality guiding features for the conditional generation of subsequent diffusion models.
[0059] Then, as Figure 3 As shown, during the forward process, the low-frequency components of the real image GT... Low-frequency components of low-resolution multispectral images LRMS The residual image is gradually subjected to noise. The forward process is a purely mathematical destruction process. It gradually adds Gaussian noise to the clean residual between the real image and the low-frequency component of the multispectral image according to a preset ratio until it becomes pure noise. This is used as training data for the neural network to learn inverse denoising.
[0060] In the reverse process, guided by the fusion features output by the MIM, a high-quality fusion result is gradually restored in the low-frequency domain. The conditional denoising process can be expressed as follows:
[0061] Among them, condition information That is, the output of the feature interaction module and The low-frequency components guide the model to recover the low-frequency details and texture information of the high-resolution multispectral image HRMS. The conditional denoising process is a parameterized Markov chain, where the probability represents the neural network's probability estimate of the inverse denoising trajectory. The final high-quality fusion result is the residual of the low-frequency components of the high-resolution multispectral image HRMS. The final image is obtained by superimposing the low-frequency signal of LRMS. .
[0062] The training objective of conditional diffusion models is typically to predict noise. Parameter optimization is achieved by minimizing the mean square error between the actual noise and the predicted noise. The loss function is defined as follows:
[0063] in, It is real Gaussian noise. This loss function is used to predict noise for the model. By continuously optimizing this loss function, the model can gradually learn the data distribution and the relationships between constraints.
[0064] In step S4 of this embodiment, the high-frequency processing branch takes the high-frequency sub-bands (HH, HL, LH) after the PAN and MS images are decomposed by discrete wavelet transform (DWT) as input, and achieves deep fusion of high-frequency features through attention guidance and multi-scale refinement.
[0065] For the high-frequency subbands of PAN, K and V are generated through normalization and convolution operations, respectively; for the high-frequency subbands of MS, Q is generated using the same method. Q and K are normalized using SoftMax and then multiplied element-wise to obtain the attention weight map S, which is then multiplied with V to achieve cross-modal attention feature extraction with MS as the query and PAN as the key, enabling the high-frequency components of MS to adaptively extract spatial details from PAN. Subsequently, the attention output is convolved and mapped, and the residual is added to the original MS high-frequency features. This residual is then fed into two cascaded MultiscaleBlocks for multi-scale refinement, fully capturing texture and edge information under different receptive fields. The refined result is then fused with another feature output from the PAN attention through Conv1×1 convolution, PReLU activation function, and Conv3×3 convolution. Finally, the original MS high-frequency residual is superimposed through convolution and addition operations to output the enhanced HH, HL, and LH high-frequency subbands. The overall design focuses on cross-modal attention and multi-scale residual refinement, which accurately injects the high-frequency spatial details of PAN into the fusion result while preserving the MS spectral structure.
[0066] like Figure 4 As shown, taking any single high-frequency band as an example (denoted as ). and The interaction and integration process is as follows: First, an improved dual cross attention mechanism is used to perform cross-modal attention calculation, with group-normalized (GN) MS features as the query and PAN features as the key and value:
[0067]
[0068] Where W represents the convolution weights, For learnable residual scaling factors, This is the scaling factor.
[0069] For the high-frequency components of the input panchromatic image PAN and low-resolution multispectral images The extracted cross-modal attention features are respectively representing .
[0070] Then, the three features mentioned above are sequentially input into the multi-scale purification module. The multi-scale purification module takes a single feature map as input and achieves rich multi-scale feature extraction and adaptive fusion through parallel multi-receptive field convolution and channel attention mechanism.
[0071] Specifically, such as Figure 5 As shown, the input features are simultaneously fed into three parallel branches: the first branch uses standard Conv3×3 convolution to capture local texture information; the second branch also uses Conv3×3 to extract medium-scale features; and the third branch uses dilated Conv3×3 convolution to expand the receptive field and capture a wider range of contextual information. The outputs of all three branches are concatenated after PReLU activation, resulting in a multi-scale joint representation. The concatenated features are then processed sequentially through Conv3×3, PreLU, and Conv1×1 for channel compression and nonlinear transformation, before being fed into a channel attention branch: pooling and activation functions generate important weights for each channel, which are then multiplied element-wise with the main features to achieve adaptive selection of features at different scales. Finally, the weighted features are summed with the residuals of the original input to output the enhanced feature map. The overall design, through parallel extraction of multiple receptive fields, channel attention weighting, and residual connections, effectively enhances the model's ability to express multi-scale texture and structural information while maintaining stable network training.
[0072] For the input Features are extracted in parallel using 3x3, 5x5, and dilated convolutions, combined with channel attention (CA) weights. Purification:
[0073]
[0074] In the formula, For feature fusion and channel dimensionality reduction operations, This is the channel attention mapping function.
[0075] Finally, in the cross-band interaction mechanism, to ensure the synergy of high-frequency information in multiple directions, the high-frequency features in the LH, HL, and HH directions are concatenated, and a global correction tensor Call is predicted through a nonlinear convolutional network. This tensor is then split into three independent tensors and weighted to their respective frequency bands.
[0076] In the formula, This represents the high-frequency correction weighting coefficient, used to control the correction strength of the global correction tensor for the high-frequency characteristics of each frequency band. Before cross-band interaction, the first Initial fusion high-frequency characteristics of each frequency band.
[0077] In step S5 of this embodiment, the final output image of step S3 is... The final output image of step S4 Performing the inverse discrete wavelet transform (IWT) yields the final output result, and the process is represented as follows:
[0078] In the formula, It is the inverse discrete wavelet transform function.
[0079] Example 2 This embodiment describes in detail the training and validation processes of the above method based on the WV3 dataset: 1. Dataset Preparation Data source: WorldView-III (WV3) satellite imagery, including two typical scenes: industrial areas and natural landscapes; Data generation: Due to the lack of real HRMS values, LRMS and PAN images were simulated and generated according to the Wald protocol (reference [1]). Specific parameters: LRMS image: resolution 128×128, number of channels S=4 (corresponding to four bands: blue, green, red, and near-infrared); PAN image: 512×512 resolution, single channel (containing only texture information); Data preprocessing: The pixel values of LRMS and PAN images were normalized to the [0,1] interval and divided into training set (80%) and test set (20%) in an 8:2 ratio to avoid overfitting during training.
[0080] 2. Model Training Steps Model initialization: Load the full-color sharpening model structure, which includes a high-frequency fusion module (128 hidden channels, 4 attention heads, 2 layers) and a low-frequency diffusion module (Conditional U-Net + diffusion model, diffusion time step T=1000, linear noise scheduling β∈[1e-4,2e-2], DDIM sampling steps 50); normalize all input images by dividing by 2047 to the [0,1] interval; fix the random seed to ensure reproducibility.
[0081] Parameter configuration: The AdamW optimizer is used, and independent learning rate parameter groups are set for high-frequency modules and low-frequency U-Net. The weight decay is 1e-4. The maximum norm of gradient clipping is 1.0. The batch size is 4, and the total number of training epochs is 700, which is divided into three course-style training phases (Phase 1: epochs 1-200, Phase 2: epochs 201-400, and Phase 3: epochs 401-700).
[0082] Training cycle: Input the LRMS and PAN images from the training set, and decompose them into low-frequency and three high-frequency sub-elements using discrete wavelet transform. Phase 1 Low-frequency training: Fix the high-frequency module and train only the low-frequency diffusion U-Net; use the residuals of GT_LL and lrms_LL as the diffusion target, extract the cross-modal conditional features of PAN and LRMS through the MIM module, guide U-Net to gradually denoise in the wavelet low-frequency domain, and predict the fused low-frequency components; set the learning rate to the initial value 2e-4. Phase 2 High-Frequency Pre-training: Fix the low-frequency module and train only the high-frequency fusion module; perform cross-modal attention fusion and multi-scale refinement on the three high-frequency subbands of PAN and LRMS (HL, LH, HH) to output the predicted high-frequency subbands; adjust the learning rate to 1e-4. Phase 3 (End-to-End Joint Fine-Tuning): Unfreeze all modules and jointly optimize the low-frequency and high-frequency branches; reconstruct the complete HRMS image by inverse transformation of the predicted low-frequency components and predicted high-frequency subbands; reduce the learning rate to 5e-6; The calculation includes a combined loss function (low-frequency diffusion loss, high-frequency structure loss, and full-image reconstruction loss), and the model parameters are updated through backpropagation. The weights of each loss are dynamically adjusted in stages. Model saving: Periodic checkpoints are saved every 50 rounds; at the same time, the total training loss is continuously tracked, and the optimal model weights are saved when the total loss is lower than the historical best value to ensure that the model's best convergence state is preserved; training logs and various loss sub-indicators are synchronously recorded to a CSV file for easy process analysis.
[0083] 3. Reasoning Steps Inference loop: Input LRMS and PAN images of the test set. For samples whose image size does not exceed the block size, perform full-image inference directly. For those that exceed the block size, use a Gaussian weighted sliding window strategy to perform block inference and then stitch them together to avoid block artifacts at the stitching boundary. The processing flow for each image block is as follows: Perform wavelet transform on the LRMS image block to decompose the low-frequency components.
[0084] High-frequency branch: The PAN and LRMS image blocks are fed into the high-frequency branch, and then undergo spatial modulation, DWT decomposition of high-frequency subbands, cross-modal attention fusion and multi-scale refinement, and cross-band correction in sequence to output the three predicted high-frequency subbands.
[0085] Low-frequency branch: Normalize PAN and LRMS to [ After [1,1], guided by the cross-modal features extracted by the MIM module and lrms_LL as a condition, the low-frequency residual is predicted by gradually denoising through DDIM sampling (50 steps); the residual is added to lrms_LL and clamped to [0,1] to obtain the fused low-frequency component.
[0086] Inverse wavelet reconstruction: The pred_LL and three predicted high-frequency subbands are inversely transformed to reconstruct the complete HRMS image block.
[0087] Each image patch is multiplied by a Gaussian weight mask and then accumulated into the full image buffer. Finally, it is divided by the weight mask to obtain a seamlessly fused complete HRMS image.
[0088] Indicator calculation and result saving: Resolution reduction with parameters test: The reconstructed image and ground truth (GT) are restored to the original DN value range, and four parameters, PSNR, SSIM, ERGAS, and SAM, are calculated.
[0089] Full-resolution parameterless test: A sliding window strategy with tile_size=256 and overlap=64 is used to infer the full-resolution image and calculate three parameterless indicators: D_lambda (spectral distortion), D_s (spatial distortion), and HQNR (overall quality). The results are saved to metrics.csv. For both tests, the 4th, 2nd, and 1st bands of the reconstructed image are extracted as RGB images and saved to the output directory for easy visual evaluation.
[0090] 4. Effect Verification The quantitative results on the WV3 test set are shown in Table 1 below.
[0091] Table 1
[0092] Experiments on the WV3 (WorldView-III), GF2 (Gaofen-2), and QB (QuickBird) datasets show that: The PSNR (peak signal-to-noise ratio) is 0.6-0.8 dB higher than that of Pandit; SSIM (structural similarity) is 0.005-0.007 higher than Pandit; SAM (spectral angle mapper) is 0.16° lower than Pandit; ERGAS (relative global error) is 0.33 lower than Pandit.
[0093] Qualitative results such as Figure 6 As shown, Figure 6 (a) is the original image. Figure 6 (j) is the HRMS image of the method of the present invention. Figures (b)-(i) are, in order, the HRMS images of existing GS method, CNMF method, IHS method, PNN method, MSDCNN method, difpan method, crossdif method, and pandit method. Figure 6As can be seen from the image, in the HRMS image (our column) generated by this invention, the roofline boundary lines of buildings are sharp and clear, without obvious blurring or stepped jagged edges (high spatial fidelity), and the road texture is highly consistent with the ground truth GT image; the hue of the vegetation area (green area in the lower right corner of the image) is consistent with GT, and there is no color shift phenomenon commonly seen in traditional methods such as IHS and GS (less spectral distortion); the geometric outline and shadow details of the building complex are completely preserved in the magnified area of the green frame, and the overall visual quality is closest to GT.
[0094] Error Map Analysis: In the full-map error heatmap (third row) and the local magnified error map (fourth row), the error map of this invention (our column) is generally dark blue, with very few high-error red and yellow pixels. In contrast, the error maps of IHS and GS show large areas of orange-red high-error regions, and difpan and crossdif show obvious checkerboard-like high-error patches in densely built-up areas (within the green box). The residual error of this invention is mainly dispersed in low-texture flat areas (such as large areas of bare ground and uniform roofs), while the error in high-frequency detail areas such as urban building edges and road intersections is close to zero, meeting the application requirements of fine remote sensing interpretation.
[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A multispectral panchromatic sharpening method based on a wavelet transform diffusion model, characterized in that: The method includes: S1. Acquire remote sensing satellite image datasets, including ground truth images (GT), panchromatic images (PAN), and low-resolution multispectral images (LRMS), and perform preprocessing. S2. Decompose the panchromatic image PAN, the low-resolution multispectral image LRMS, and the real image GT by discrete wavelet transform, and extract their low-frequency and high-frequency components respectively. S3. Construct a low-frequency processing branch, in which the panchromatic image PAN and the low-resolution multispectral image LRMS are first input into the feature interaction module to extract fusion condition features. Then, a forward noise addition process is constructed by adding noise to the low-frequency components of the real image GT and the low-resolution multispectral image LRMS, and a reverse noise reduction process is constructed guided by the fusion condition features output by the feature interaction module to obtain the final output low-frequency features. S4. Construct a high-frequency processing branch, in which the high-frequency features of the low-resolution multispectral image LRMS are used as query Q, and the high-frequency components of the panchromatic image PAN are used as key K and value V to perform cross-modal attention calculation. Multi-scale feature extraction is performed through a multi-scale purification module for adaptive fusion to obtain the final output high-frequency features. S5. Perform inverse discrete wavelet transform (IWT) on the obtained low-frequency and high-frequency features to reconstruct the final multispectral panchromatic sharpened image.
2. The multispectral panchromatic sharpening method based on a wavelet transform diffusion model according to claim 1, characterized in that: In step S1, real image GT, panchromatic image PAN, and low-resolution multispectral image LRMS are obtained from publicly available remote sensing data resources. All image data are preprocessed using data normalization to unify the range of image pixel values and complete data standardization.
3. The multispectral panchromatic sharpening method based on a wavelet transform diffusion model according to claim 1, characterized in that: In step S2, for any two-dimensional image signal After discrete wavelet transform decomposition, one low-frequency approximate component and three high-frequency detail components are obtained, which are represented as follows: in, It represents low-frequency components, preserving the overall outline and brightness information of the image; The high-frequency components in the horizontal direction correspond to the vertical edge details; The high-frequency components in the vertical direction correspond to the horizontal edge details; These are the high-frequency components in the diagonal direction, corresponding to texture details; For the input real image GT, panchromatic image PAN, and low-resolution multispectral image LRMS, the decomposed low-frequency and high-frequency components are represented as follows: , , .
4. The multispectral panchromatic sharpening method based on a wavelet transform diffusion model according to claim 3, characterized in that: In step S3, during the forward process, the low-frequency components of the real image GT are... Low-frequency components of low-resolution multispectral images LRMS Noise is gradually added to the residual image. In the forward process, Gaussian noise is gradually added to the clean residual between the real image and the low-frequency component of the multispectral image according to a preset ratio until it becomes pure noise. This is used as training data for the neural network to learn inverse denoising. In the reverse process, guided by the conditional fusion features output by the feature interaction module, a high-quality fusion result is gradually restored in the low-frequency domain. Its conditional denoising process can be represented as follows: Among them, condition information That is, the output of the feature interaction module and The low-frequency components guide the model to recover the low-frequency details and texture information of the high-resolution multispectral image HRMS. The conditional denoising process is a parameterized Markov chain, where the probability represents the neural network's probability estimate of the inverse denoising trajectory. The final high-quality fusion result is the residual of the low-frequency components of the high-resolution multispectral image HRMS. The final image is obtained by superimposing the low-frequency signal of LRMS. .
5. The multispectral panchromatic sharpening method based on a wavelet transform diffusion model according to claim 4, characterized in that: The feature interaction module in step S3 includes a channel interaction module and a spatial-level interaction module, wherein, Channel interaction module receives and As input, global average pooling and max pooling operations are performed on both respectively, compressing the spatial dimension into channel descriptors, which are then concatenated to obtain... global vector : Then, channel attention weights for LRMS and PAN are generated separately via MLP. and The two sets of weights are multiplied element-wise with the original input and then summed to achieve a weighted sum of cross-modal channel-level features, outputting... and : The spatial-level interaction module receives the output of the first stage. and splicing the two together The combined features, while through average pooling, Convolution, max pooling, and three other paths extract spatial statistics. : Spatial attention masks are then generated using MLP. and The spatial masks are applied to their respective input features, and cross-modal spatial feature selection and enhancement are achieved through element-wise multiplication and residual addition, ultimately outputting high-quality features. and : in, This represents the Sigmoid activation function. This indicates element-wise multiplication.
6. The multispectral panchromatic sharpening method based on a wavelet transform diffusion model according to claim 4, characterized in that: In step S3, the training objective of the conditional diffusion model is to use noise prediction. Parameter optimization is achieved by minimizing the mean square error between the actual noise and the predicted noise. The loss function is defined as follows: in, It is real Gaussian noise. The model predicts noise; by continuously optimizing this loss function, the model can gradually learn the data distribution and conditional constraints.
7. The multispectral panchromatic sharpening method based on a wavelet transform diffusion model according to claim 1, characterized in that: In step S4, the high-frequency processing branch takes the high-frequency sub-bands of the PAN and LRMS images after discrete wavelet transform decomposition as input, and achieves deep fusion of high-frequency features through attention guidance and multi-scale refinement. The process includes: For the high-frequency subbands of PAN, K and V are generated through normalization and convolution operations, respectively; for the high-frequency subbands of LRMS, Q is generated using the same method. Q and K are normalized using SoftMax and then multiplied element-wise to obtain the attention weight map S. This S is then multiplied by V to extract cross-modal attention features using LRMS as the query and PAN as the key. Subsequently, the attention output is convolved and then added to the residual of the original LRMS high-frequency features to obtain the cross-attention features. in, Represents the convolution weights. For learnable residual scaling factors, The scaling factor is used to calculate the high-frequency components of the input panchromatic image PAN. and low-resolution multispectral images The extracted cross-modal attention features are respectively representing ; The obtained features are input into two cascaded multi-scale refinement modules for multi-scale refinement. The refinement results are then fused with another feature output through PAN attention by Conv1×1 convolution, activation function PReLU, and Conv3×3 convolution. Finally, the original LRMS high-frequency residuals are superimposed through convolution and addition operations to output the enhanced HH, HL, and LH high-frequency subbands.
8. The multispectral panchromatic sharpening method based on a wavelet transform diffusion model according to claim 7, characterized in that: The input features of the multi-scale purification module in step S4 are simultaneously fed into three parallel branches: the first branch uses standard Conv3×3 convolution to capture local texture information; the second branch also uses Conv3×3 to extract medium-scale features. The third branch uses Conv3×3 dilated convolutions to expand the receptive field and capture a wider range of contextual information. The outputs of all three branches are concatenated after PReLU activation, resulting in a multi-scale joint representation. The concatenated features are then processed sequentially through Conv3×3, PreLU, and Conv1×1 for channel compression and nonlinear transformation, before being fed into a channel attention branch. This branch generates important weights for each channel through pooling and activation functions, multiplying them element-wise with the main features to achieve adaptive selection of features at different scales. Finally, the weighted features are added to the original input as residuals to output an enhanced feature map. The process is represented as follows: In the formula, For feature fusion and channel dimensionality reduction operations, For channel attention mapping function; Finally, the high-frequency features in the LH, HL, and HH directions are concatenated, and the global correction tensor Call is predicted through a nonlinear convolutional network. This tensor is then split into three independent tensors and weighted to their respective frequency bands. In the formula, This represents the high-frequency correction weighting coefficient, used to control the correction strength of the global correction tensor for the high-frequency characteristics of each frequency band. Before cross-band interaction, the first Initial fusion high-frequency characteristics of each frequency band.
9. The multispectral panchromatic sharpening method based on a wavelet transform diffusion model according to claim 1, characterized in that: In step S5, the final output image from step S3 is... The final output image of step S4 Performing the inverse discrete wavelet transform (IWT) yields the final output result, and the process is represented as follows: In the formula, It is the inverse discrete wavelet transform function.
10. A multispectral panchromatic sharpening method based on a wavelet transform diffusion model according to claim 1, characterized in that: The training process for the method consists of three training phases, including: Phase 1 Low-frequency pre-training: Fix the high-frequency processing branch and train only the low-frequency processing branch; Phase 2 High-Frequency Pre-training: Fix the low-frequency processing branch and train only the high-frequency processing branch; Phase 3 end-to-end joint fine-tuning: Unfreeze all branches, jointly optimize low-frequency and high-frequency branches; reconstruct the complete HRMS image by inverse transformation of the predicted low-frequency components and predicted high-frequency subbands; The combined loss function, which includes low-frequency diffusion loss, high-frequency structure loss, and full-image reconstruction loss, is calculated. The model parameters are updated through backpropagation, and the weights of each loss are dynamically adjusted in stages.