A panchromatic sharpening method based on high-order state space modeling

Abstract

The application discloses a panchromatic sharpening method based on high-order state space modeling, introduces a hierarchical state modeling mechanism and a multi-stage joint learning framework, effectively improves the spatial detail restoration and spectral fidelity of a remote sensing image, and has good performance stability, modeling capability and practical application value. A deep neural network architecture for multispectral and panchromatic image fusion is constructed, low-resolution multispectral images are first upsampled to match the spatial dimension of the panchromatic image, then spatial and spectral multi-scale features are extracted through a multi-stage feature extraction module, then a fusion module based on high-order structured state space modeling is introduced, the module constructs a state update process with long-range dependency modeling capability, information integration and image restoration are realized through a cross-stage residual connection and a feature reconstruction module, and semantic alignment and high-order interaction between the multispectral and panchromatic modalities are realized through a cross-modal fusion module, so that the generalization performance of the model in a complex scene is enhanced.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing and image fusion technology, and in particular to a panchromatic sharpening method based on high-order state space modeling. Background Technology

[0002] With the rapid development of remote sensing imaging technology, remote sensing satellites are widely used in various fields such as resource surveys, urban planning, and environmental monitoring. Multispectral (MS) images have rich spectral information but low spatial resolution, while panchromatic (PAN) images have high spatial resolution but lack spectral resolution. In order to acquire both high spatial and hyperspectral information simultaneously, image fusion technology, especially pan-sharpening, has become a research hotspot.

[0003] Traditional image fusion methods, such as Principal Component Analysis (PCA), Intensity-Hue-Saturation Transform (IHS), Brovey Transform, and Generalized Gaussian Filtering, while computationally efficient, often suffer from spectral distortion or insufficient spatial detail. In recent years, with the development of deep learning technology, Convolutional Neural Networks (CNNs) and Transformer architectures have been widely introduced into the field of image fusion, significantly improving the quality of fused images. However, CNN structures are limited by their receptive field, making it difficult to model long-range dependencies; while Transformers, although possessing global modeling capabilities, face challenges in resource-constrained scenarios due to their high computational cost.

[0004] Furthermore, state-space models (SSMs), as a modeling framework with continuous modeling capabilities and linear time complexity, have shown great promise in image modeling, time series analysis, and other fields. However, most existing SSM-based methods remain at the first-order or simplified structure level and have not yet effectively explored their potential for higher-order spatial-spectral interactions in image fusion tasks.

[0005] Therefore, there is an urgent need for a novel remote sensing image fusion method that can maintain spectral consistency while taking into account both spatial detail enhancement and computational efficiency, so as to meet the dual requirements of fused image quality and model deployability in practical applications. Summary of the Invention

[0006] In view of the limitations of existing remote sensing image fusion methods, such as limited spatial feature modeling capabilities, high computational complexity, and insufficient multimodal interaction modeling, this invention provides a panchromatic sharpening method based on high-order state-space modeling. This method addresses the issues of insufficient spatial-spectral information interaction, limited remote dependency modeling, and low model deployment efficiency in the fusion of multispectral and panchromatic images. By introducing a high-order state-space modeling mechanism and a recursive phased structure, this invention effectively improves the spatial detail and spectral preservation capabilities of the fused image, while also possessing good engineering deployability, meeting the dual requirements of accuracy and efficiency in remote sensing image analysis.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] In a first aspect, the present invention provides a panchromatic sharpening method based on high-order state space modeling, comprising the following steps:

[0009] Step 1: Acquire pairs of remote sensing images with different resolutions, including low-resolution multispectral images and high-resolution panchromatic images. Upsample the multispectral images to make their resolution consistent with that of the panchromatic images, and obtain the upsampled multispectral images.

[0010] Step 2: Process the upsampled multispectral image I. MS Compared with the original panchromatic image I PAN Shallow feature extraction is performed to obtain its initial feature representation, denoted as F. MS With F PAN ;

[0011] Step 3: Construct a high-order state space modeling module. Input the extracted features into the high-order state space modeling module. The high-order state space modeling module uses structured state space units to jointly model spatial and channel information.

[0012] Step 4: To enhance the multispectral image I MS With panchromatic image I PAN Cross-modal information interaction between them introduces a cross-modal fusion path, including a channel exchange module and a cross-modal fusion module, to achieve modal-level dynamic fusion and high-order structure modeling;

[0013] Step 5: Input the fused features into the reconstruction module and output a high-resolution multispectral image.

[0014] In one embodiment of the present invention, step 1 specifically includes:

[0015] Step 1.1: Acquire high-resolution multispectral images and their corresponding panchromatic images, and perform cropping operations to construct an image dataset; the cropping operation obtains pairs of remote sensing images with different resolutions, including low-resolution multispectral images. and high-resolution panchromatic images Where H represents the height of the high-resolution panchromatic image, W represents the width of the high-resolution panchromatic image, and c represents the number of channels of the high-resolution panchromatic image; the corresponding h is the height of the low-resolution multispectral image, which is set here as h = H / 4, w is the width of the low-resolution multispectral image, w = W / 4, and C represents the number of channels of the low-resolution multispectral image.

[0016] Step 1.2: Upsample the low-resolution multispectral image to match the resolution of the high-resolution panchromatic image using interpolation methods to obtain the upsampled multispectral image.

[0017] In one embodiment of the present invention, step 2 specifically includes:

[0018] Step 2.1: Process the input image using a two-dimensional convolution operator with learnable weight parameters. The kernel size is 3×3, the stride is 1, and the padding is 1 to maintain the output size. The result of the convolution operation is expressed as:

[0019] F (1) =W conv *I+b conv

[0020] Where * represents the convolution operation, W conv b represents the kernel weights. conv For bias terms;

[0021] Step 2.2: To enhance the numerical stability of the model under different image samples, the instance normalization module is used to normalize the convolutional features. The calculation formula is as follows:

[0022]

[0023] Where μ and σ 2 denoted as the mean and variance for each channel, γ and β are learnable affine transformation parameters, and ∈ is a small constant to prevent division by zero;

[0024] Step 2.3: Select SiLU as the nonlinear activation function to enhance the network's representational power. The definition of SiLU is:

[0025] F (3) =SiLU(F (2) ) = F (2) ·σ(F (2) )

[0026] Where σ(·) represents the Sigmoid function;

[0027] For multispectral images I MS With panchromatic image I PAN The shallow feature extraction process is represented as follows:

[0028] F MS =SiLU(InstanceNorm(Conv 3×3 (I MS )))

[0029] F PAN =SiLU(InstanceNorm(Conv 3×3 (I PAN ))).

[0030] In one embodiment of the present invention, step 3 specifically includes:

[0031] Step 3.1: Design three independent state evolution paths, each corresponding to a feature subspace of different scales or semantic dimensions, with channel dimensions of D / 4, D / 2 and D, respectively, to express the spatial context modeling capability from shallow to deep and from local to global.

[0032] Let the input feature tensor be Where L = H × W is the number of spatial positions after flattening, and D is the number of channels. The specific modeling process is as follows:

[0033] Step 3.1.1, Channel Dimension Upgrading and Partitioning: First, perform a linear transformation on the input feature X to increase its dimension to 2D, and then divide it into three groups of sub-features according to the proportions:

[0034]

[0035] in, These represent the feature subspaces corresponding to the three sub-paths;

[0036] Step 3.1.2, Three-way state evolution modeling: Each path uses an independent state-space evolution model, combining depthwise convolution and the SS2D operator to model its sequential state evolution process:

[0037]

[0038] Among them, DWConv i (·) indicates a depthwise separable convolution operation in the i-th branch, used for local feature enhancement; SS2D i (·) represents a two-dimensional state-space modeling module, used to capture long-distance dependencies between spatial locations;

[0039] SS2D introduces a state-space-based dynamic modeling mechanism that captures long-range dependencies and temporal context information in remote sensing image sequences through the temporal evolution of latent states. This mechanism can be viewed as a modeling framework based on linear dynamic systems, and its continuous-time form is expressed as follows:

[0040] h′(t)=Ah(t)+Bx(t), y(t)=Ch′(t)

[0041] in, Indicates the current input signal. This indicates the corresponding output response. The hidden state vector. Here is the state transition matrix. For the input mapping matrix, To output the mapping matrix;

[0042] To meet the demands of computers processing discrete time series data, a zero-order hold strategy is introduced to discretize the above continuous model, resulting in a discrete state transition matrix and an input mapping matrix:

[0043]

[0044] Where Δ is a fixed time step and I represents the identity matrix; subsequently, sequence modeling is performed based on the discrete state-space model:

[0045]

[0046] y t =Ch t

[0047] This modeling process is equivalent to a structured convolution process, and its kernel function is defined as:

[0048]

[0049] Where * represents a one-dimensional convolution operation, and L represents the length of the input sequence.

[0050] In one embodiment of the present invention, step 4 specifically includes:

[0051] Step 4.1, Channel Exchange Cross-fusion: Before entering the fusion stage, intermediate feature tensors from the multispectral path and the panchromatic path are extracted respectively. The channel exchange module is used to perform cross-fusion in the channel dimension. The specific operation is as follows:

[0052]

[0053] By selecting some channels from the PAN features to replace the corresponding channels in the MS features, or by selecting some channels from the MS features to replace the corresponding channels in the PAN features, the initial injection and interaction of complementary structural information can be achieved.

[0054] Step 4.2, Cross-modal fusion module: To further model cross-modal high-order representation information, a guided enhancement mechanism is adopted, which guides the fusion of panchromatic channel features by information-rich multispectral features. Specifically, this includes:

[0055]

[0056] The cross-fused features are fed into the cross-modal fusion module for further modeling. The MS channel features, which have richer information content, are used as guides. After linear mapping and SiLU activation, they are subjected to element-wise Hadamard product operation with the PAN channel features to realize the fusion guidance mechanism and enhance the collaborative expression between modalities.

[0057] In one embodiment of the present invention, step 5 specifically includes:

[0058] Step 5.1: Input the fused features into the image reconstruction module, which contains a 3×3 convolutional layer, to generate the output high-resolution multispectral image HrMS.

[0059]

[0060] Step 5.2: Introduce upsampled image MS through skip connections to improve detail restoration capabilities:

[0061]

[0062] In one embodiment of the present invention, the method further includes: step 6, training the entire network structure through supervised learning, wherein the loss function includes a spatial error term, a structural similarity term, and a spectral consistency term, to achieve dual optimization of image sharpness and spectral fidelity; step 6 specifically includes:

[0063] Step 6.1, Spatial Domain Error Term: This term measures the overall error between the reconstructed image and the reference image in pixel space, using the mean absolute error as the metric.

[0064]

[0065] Among them, I HrMS To output an image to the network, I GT This is the corresponding high-resolution multispectral reference image;

[0066] Step 6.2, Spectral Consistency Term: To maintain the spectral consistency of the fused image, a spectral angle mapping or scale-free relative global error metric is introduced as an additional regularization term to control spectral angle deviation and global error, using SAM:

[0067]

[0068] Among them, I HrMS To output an image to the network, I GT This is the corresponding high-resolution multispectral reference image;

[0069] Step 6.3: The final combination of multinomial loss functions is a weighted sum, defined as follows:

[0070]

[0071] Among them, λ1 and λ2 are hyperparameters used to balance the impact weights of various losses.

[0072] Secondly, the present invention provides a full-color sharpening system based on high-order state space modeling, which uses the method described above, including:

[0073] The image preprocessing module is used to acquire pairs of remote sensing images with different resolutions, including low-resolution multispectral images and high-resolution panchromatic images, and to upsample the multispectral images to make their resolution consistent with that of the panchromatic images, thus obtaining the upsampled multispectral images.

[0074] The feature extraction module is used to extract features from the upsampled multispectral image I. MS Compared with the original panchromatic image I PAN Shallow feature extraction is performed to obtain its initial feature representation, denoted as F. MS With F PAN ;

[0075] A high-order state space modeling module construction module is used to construct a high-order state space modeling module. The extracted features are input into the high-order state space modeling module, which uses structured state space units to jointly model spatial and channel information.

[0076] The modal-level dynamic fusion and high-order structure modeling module is used to enhance multispectral image I... MS With panchromatic image I PAN Cross-modal information interaction between them introduces a cross-modal fusion path, including a channel exchange module and a cross-modal fusion module, to achieve modal-level dynamic fusion and high-order structure modeling;

[0077] The high-resolution multispectral image output module is used to input the fused features into the reconstruction module and output a high-resolution multispectral image.

[0078] Thirdly, the present invention provides a computer-readable storage medium storing computer instructions which are executed by a processor using the method described above.

[0079] Fourthly, the present invention provides a computer program product, the computer program product storing computer instructions, the computer instructions being executed by a processor using the method described above.

[0080] The beneficial effects achieved by this invention are as follows:

[0081] This invention provides a panchromatic sharpening method based on high-order state space modeling. By introducing a hierarchical state modeling mechanism and a multi-stage joint learning framework, it effectively improves the spatial detail restoration and spectral fidelity of remote sensing images, exhibiting good performance stability, modeling capabilities, and practical application value. This method constructs a deep neural network architecture for multispectral and panchromatic image fusion. First, low-resolution multispectral images are upsampled to match the spatial dimension of the panchromatic image. Then, a multi-stage feature extraction module extracts spatial and spectral multi-scale features. Next, a fusion module based on high-order structured state space modeling is introduced. This module employs one-dimensional convolution, gating mechanisms, and selective scanning mechanisms to construct a state update process with long-range dependency modeling capabilities. Furthermore, information integration and image restoration are achieved through cross-stage residual connections and feature reconstruction modules. In addition, a cross-modal fusion module (C-PHoM) is proposed to achieve semantic alignment and high-order interaction between multispectral and panchromatic modalities, enhancing the model's generalization performance in complex scenes. Therefore, the fusion quality of this invention is higher: it can effectively enhance spatial details and maintain spectral consistency; it has stronger modeling capabilities: based on a high-order state modeling mechanism, it can mine long-range spatial dependencies in images; it has better computational efficiency: it adopts a state-space structure with linear complexity, which is suitable for large-scale remote sensing image fusion tasks; and it has wider adaptability: it supports the fusion of various remote sensing images (such as WorldView-II, GF2, WV3, etc.) and has good prospects for engineering applications. Attached Figure Description

[0082] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the drawings without creative effort.

[0083] Figure 1 This is an overall flowchart of the method of the present invention.

[0084] Figure 2This is a diagram of the Mamba module framework used in the method of the present invention.

[0085] Figure 3 This is a framework diagram of the high-order state-space modeling module used in the method of the present invention.

[0086] Figure 4 The images show the subjective experimental results of the method of this invention, the traditional pancolor sharpening method, and the deep learning pancolor sharpening method on the WorldView-II dataset.

[0087] Figure 5 The image shows the error comparison between the method of this invention and traditional pancolor sharpening methods and deep learning pancolor sharpening methods on the WorldView-II dataset.

[0088] Figure 6 The images show the subjective experimental results of the method of this invention, the traditional pancolor sharpening method, and the deep learning pancolor sharpening method on the Gaofen-2 dataset.

[0089] Figure 7 The figure shows the error experiment results of the method of this invention, the traditional pancolor sharpening method, and the deep learning pancolor sharpening method on the Gaofen-2 dataset;

[0090] Figure 8 The images show the subjective experimental results of the method of this invention, the traditional pancolor sharpening method, and the deep learning pancolor sharpening method on the WorldView-III dataset.

[0091] Figure 9 The image shows the error results of the method of this invention compared with the traditional pancolor sharpening method and the deep learning pancolor sharpening method on the WorldView-III dataset. Detailed Implementation

[0092] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0093] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the invention, are intended to cover non-exclusive inclusion.

[0094] In the description of the embodiments of this invention, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this invention, "multiple" means two or more, unless otherwise explicitly defined.

[0095] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least some embodiments of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0096] In the description of the embodiments of this invention, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0097] like Figures 1 to 3 As shown, this invention provides a panchromatic sharpening method based on high-order state space modeling, comprising the following steps:

[0098] Step 1: Input image preparation, including data acquisition and preprocessing; Step 1 specifically includes:

[0099] Step 1.1: Acquire high-resolution multispectral images and their corresponding panchromatic images, and perform cropping operations to construct an image dataset; the cropping operation obtains remote sensing image pairs with different resolutions, including low-resolution multispectral images (LRMS). And high-resolution panchromatic images (PAN) Where H represents the height of the high-resolution panchromatic image, W represents the width of the high-resolution panchromatic image, and c represents the number of channels of the high-resolution panchromatic image, which is 1 by default; the corresponding h is the height of the low-resolution multispectral image, which is set to h = H / 4 here, w is the width of the low-resolution multispectral image, w = W / 4, and C represents the number of channels of the low-resolution multispectral image.

[0100] Step 1.2: Upsample the low-resolution multispectral image to match the resolution of the high-resolution panchromatic image using an interpolation method (such as bicubic interpolation), thus obtaining the upsampled multispectral image.

[0101] Step 2: Process the upsampled multispectral image I MS Compared with the original panchromatic image IPAN Shallow feature extraction is performed to obtain its initial feature representation, denoted as F. MS With F PAN Step 2 specifically includes:

[0102] Step 2.1: Process the input image using a two-dimensional convolution operator with learnable weight parameters. The kernel size is 3×3, the stride is 1, and the padding is 1 to maintain the output size. The result of the convolution operation is expressed as:

[0103] F (1) =W conv *I+b conv

[0104] Where * represents the convolution operation, W conv b represents the kernel weights. conv For bias terms;

[0105] Step 2.2: To enhance the numerical stability of the model under different image samples, the instance normalization module is used to normalize the convolutional features. The calculation formula is as follows:

[0106]

[0107] Where μ and σ 2 denoted as the mean and variance for each channel, γ and β are learnable affine transformation parameters, and ∈ is a small constant to prevent division by zero;

[0108] Step 2.3: Select SiLU (Sigmoid Linear Unit) as the nonlinear activation function to enhance the network's representational power. The definition of SiLU is:

[0109] F (3) =SiLU(F (2) ) = F (2) ·σ(F (2) )

[0110] Where σ(·) represents the Sigmoid function;

[0111] In summary, for multispectral images I MS and panchromatic image I PAN The shallow feature extraction process is represented as follows:

[0112] F MS =SiLU(InstanceNorm(Conv 3×3 (I MS )))

[0113] F PAN=SiLU(InstanceNorm(Conv 3×3 (I PAN )))

[0114] Furthermore, the feature extraction module can stack multiple such operation units to form a shallow convolutional encoder with residual structures. Its role is to provide stable, efficient, and information-fidelity modal feature representations, providing high-quality input for subsequent state-space modeling and modal fusion.

[0115] Step 3: Construct a high-order state-space modeling module. The core of this invention lies in constructing a high-order state-space modeling module with multi-path dynamic modeling capabilities to more fully explore the complex spatial dependencies and semantic representation capabilities in multispectral and panchromatic images. Step 3 specifically includes:

[0116] Step 3.1: Design three independent state evolution paths, each corresponding to a feature subspace of different scales or semantic dimensions, with channel dimensions of D / 4, D / 2 and D, respectively, to express the spatial context modeling capability from shallow to deep and from local to global.

[0117] Let the input feature tensor be Where L = H × W is the number of spatial positions after flattening, and D is the number of channels. The specific modeling process is as follows:

[0118] Step 3.1.1, Channel Dimension Upgrading and Partitioning: First, perform a linear transformation on the input feature X to increase its dimension to 2D, and then divide it into three groups of sub-features according to the proportions:

[0119]

[0120] in, These represent the feature subspaces corresponding to the three sub-paths;

[0121] Step 3.1.2, Three-way state evolution modeling: Each path uses an independent state-space evolution model, combining depthwise convolution and the SS2D (Selective Scan 2D) operator to model its sequence state evolution process:

[0122]

[0123] Among them, DWConv i (·) indicates a depthwise separable convolution operation in the i-th branch, used for local feature enhancement; SS2D i (·) represents a two-dimensional state-space modeling module, used to capture long-distance dependencies between spatial locations;

[0124] SS2D introduces a state-space dynamic modeling (SSM) mechanism. Its core idea is to capture long-range dependencies and temporal context information in remote sensing image sequences through the temporal evolution of latent states. This mechanism can be viewed as a modeling framework based on linear dynamic systems, and its continuous-time form can be expressed as:

[0125] h′(t)=Ah(t)+Bx(t),y(t)=Ch′(t)

[0126] in, Indicates the current input signal. This indicates the corresponding output response. The hidden state vector. Here is the state transition matrix. For the input mapping matrix, To output the mapping matrix;

[0127] To meet the demands of computers processing discrete time series data, a zero-order hold (ZOH) strategy is introduced to discretize the aforementioned continuous model, resulting in a discrete state transition matrix and input mapping matrix:

[0128]

[0129] Where Δ is a fixed time step and I represents the identity matrix; subsequently, sequence modeling is performed based on the discrete state-space model:

[0130]

[0131] y t =Ch t

[0132] In its implementation, this modeling process is equivalent to a structured convolution process, and its kernel function is defined as:

[0133]

[0134] Where * represents a one-dimensional convolution operation and L represents the length of the input sequence; this modeling method not only has linear time complexity, but also enables long-range dependency modeling in the global scope, significantly enhancing the feature representation capability of multispectral images.

[0135] Step 4: To enhance the multispectral image I MS With panchromatic image I PANTo enhance spatial-spectral joint modeling capabilities through cross-modal information interaction, this invention introduces a cross-modal fusion path into the network structure, including a channel swapping module (ChannelSwap) and a cross-modal fusion module (C-PHoM), to achieve modal-level dynamic fusion and high-order structure modeling; step 4 specifically includes:

[0136] Step 4.1, Channel Swap Cross-fusion: Before entering the fusion stage, intermediate feature tensors from the multispectral path and the panchromatic path are extracted respectively. The channel swap module is used to perform cross-fusion in the channel dimension. The specific operation is as follows:

[0137]

[0138]

[0139] Select some channels from the PAN features to replace the corresponding channels in the MS features, and vice versa, to achieve the initial injection and interaction of complementary structural information;

[0140] Step 4.2, Cross-Modal Fusion Module (C-PHoM): To further model cross-modal high-order representation information, this module adopts a guided enhancement mechanism, which guides the fusion of panchromatic channel features by information-rich multispectral features. Specifically, it includes:

[0141]

[0142] The cross-fused features are fed into the cross-modal fusion module (C-PHoM) for further modeling. The MS channel features with richer information content are used as guides. After linear mapping and SiLU activation, they are subjected to element-wise Hadamard product operation with the PAN channel features to realize the fusion guidance mechanism and enhance the collaborative expression between modalities.

[0143] Step 5, image reconstruction; Step 5 specifically includes:

[0144] Step 5.1: Input the fused features into the image reconstruction module, which contains a 3×3 convolutional layer, to generate the output high-resolution multispectral image HrMS.

[0145]

[0146] Step 5.2: Introduce upsampled image MS through skip connections to improve detail restoration capability;

[0147]

[0148] Step 6: Train the entire network structure through supervised learning. The loss function includes spatial error terms (such as mean squared error MSE), structural similarity (SSIM), and spectral consistency terms (such as SAM or ERGAS) to achieve dual optimization of image sharpness and spectral fidelity. Step 6 specifically includes:

[0149] Step 6.1, Spatial Domain Error Term: This term measures the overall error between the reconstructed image and the reference image in pixel space. Here, the metric used is Mean Absolute Error (MAE).

[0150]

[0151] Among them, I HrMS To output an image to the network, I GT This is the corresponding high-resolution multispectral reference image;

[0152] Step 6.2, Spectral Consistency Term: To maintain the spectral consistency of the fused image, metrics such as Spectral Angle Mapper (SAM) or Scale-Free Relative Global Error (ERGAS) are introduced as additional regularization terms to control spectral angle deviation and global error. Here, SAM is used.

[0153]

[0154] Among them, I HrMS To output an image to the network, I GT This is the corresponding high-resolution multispectral reference image;

[0155] Step 6.3: The final combination of multinomial loss functions is a weighted sum, defined as follows:

[0156]

[0157] λ1 and λ2 are hyperparameters used to balance the influence weights of various losses. This composite loss function can effectively guide the model to gradually improve the detail restoration ability and spectral preservation performance of the fused image during training, and achieve synergistic optimization of spatial and spectral information.

[0158] like Figure 1 The diagram shown is an overall flowchart of the method of this invention. The input data includes an upsampled low-resolution multispectral image and a high-resolution panchromatic image. The multispectral image is first upsampled to make its size consistent with the panchromatic image.

[0159] In the feature extraction stage (Stage 1), the panchromatic and upsampled multispectral images are subjected to preliminary feature extraction through 3×3 convolution and normalization, and further mapped to a unified embedding dimension through a feature embedding module (Tokenization). Next, the proposed network introduces multiple higher-order Mamba blocks to perform deep spatial modeling of the two feature paths. This stage focuses on capturing the spatial structure and primary semantic features of the image.

[0160] Entering the feature fusion stage (Stage 2), this invention employs a Channel Swap (CS) module to perform cross-information interaction between multispectral and panchromatic features. The fused features are then fed into deeper, higher-order Mamba blocks for joint modeling. In this stage, the invention further introduces residual connection structures and the SiLU activation function to enhance feature representation capabilities while maintaining stable training. Finally, a high-resolution multispectral image (HrMS) is recovered through 3×3 convolutions and skip connections.

[0161] To further clarify the functions and composition of each key sub-module in the fusion structure proposed in this invention, the core constituent units are described in detail below, including: the basic state space modeling module (Vanilla Mamba), the higher-order state modeling module (Higher-Order Mamba Block), and the cross-modal fusion module (C-PHoM), etc.

[0162] like Figure 2 The diagram shows the basic Mamba structure (Vanilla Mamba) referenced in this invention. This module, based on state-space modeling, maps continuous-time dynamic systems to discrete update functions. The core of the module includes the state update function h′(t) = Ah(t) + Bx(t), the output function y(t) = Ch′(t), and the structured parameter matrix (A, B, C, D). It also incorporates several sub-modules such as linear mapping, embedding, normalization (LN), depthwise separable convolution (DWConv), and the SiLU activation function.

[0163] This structure has been verified to have good long-range dependency modeling capabilities in multiple modeling tasks and can be extended as the basic unit of this invention.

[0164] like Figure 3 The diagram shown is a structural diagram of the Higher-Order Mamba Block proposed in this invention. Based on Vanilla Mamba, this module adopts a multi-order state path design, that is, it sets up multiple Selective Scan 2D (SS2D) state update channels, and each channel models spatial feature representations at different granularities.

[0165] The module's internal structure consists of three layers of serial state paths, modeling spatial states with channel dimensions of D / 4, D / 2, and D respectively. Each path is constructed using SS2D+DWConv+Linear+SiLU, and multi-order state representations are fused at the output using Hadamard product. Finally, a linear mapping layer unifies the output to the original embedding dimension, achieving joint modeling of complex spatial structures and long-range dependencies.

[0166] Furthermore, to enhance cross-modal information interaction between multispectral and panchromatic images and improve spatial-spectral joint modeling capabilities, this invention introduces a cross-modal fusion path into the network structure, including a channel swapping module (ChannelSwap) and a cross-modal fusion module (C-PHoM), to achieve modal-level dynamic fusion and high-order structure modeling.

[0167] This invention introduces a ChannelSwap module into the network structure. Before entering the fusion stage, intermediate feature tensors from the multispectral path and the panchromatic path are extracted and cross-fused in the channel dimension.

[0168] This invention introduces a C-PHoM (Cross-modal PHoM) module into the network structure to enhance the fusion of MS features and PAN features at a specific stage. Specifically, by utilizing the rich information content of MS features, it introduces them into a linear transformation and SiLU activation to dominate the fusion process through Hadamard product.

[0169] This high-order module can be flexibly stacked in multiple layers and supports connections with backbone feature residuals to enhance deep modeling capabilities. It can be widely applied to visual tasks including image fusion, super-resolution, and object detection, exhibiting good generalization and engineering feasibility.

[0170] In summary, the panchromatic sharpening method proposed in this invention, based on high-order state space modeling, effectively improves the spatial detail restoration and spectral fidelity of remote sensing images by introducing a hierarchical state modeling mechanism and a multi-stage joint learning framework. It exhibits good performance stability, modeling capabilities, and practical application value. This method constructs a deep neural network architecture for multispectral and panchromatic image fusion. First, low-resolution multispectral images are upsampled to match the spatial dimension of the panchromatic image. Then, a multi-stage feature extraction module extracts spatial and spectral multi-scale features. Next, a fusion module based on high-order structured state space modeling is introduced. This module employs one-dimensional convolution, gating mechanisms, and selective scanning mechanisms to construct a state update process with long-range dependency modeling capabilities. Furthermore, information integration and image restoration are achieved through cross-stage residual connections and feature reconstruction modules. In addition, a cross-modal fusion module (C-PHoM) is proposed to achieve semantic alignment and high-order interaction between multispectral and panchromatic modalities, enhancing the model's generalization performance in complex scenarios.

[0171] The effects of this invention can be further illustrated by the following simulation experiments.

[0172] 1. Simulation conditions and parameters

[0173] In the simulation experiments of this invention, three remote sensing image datasets—WorldView-II (WV2), Gaofen2 (GF2), and WorldView-III (WV3)—were selected. All datasets underwent simulated degradation using the Wald protocol to construct a referenceless fusion evaluation scenario. The image spatial resolution ratio was 1:4, the input multispectral image size was 32×32×4, and the panchromatic image size was 128×128×1. The experiments were conducted on a single NVIDIA 3090Ti GPU using the PyTorch framework. Training was performed for a total of 200, 500, and 500 iterations respectively, with an initial learning rate of 5×10⁻⁶. -4 The optimizer is Adam, and the batch size is 4.

[0174] 2. Simulation Content and Result Analysis

[0175] In the simulation experiment, to verify the effectiveness and advancement of the pan-color sharpening method based on high-order state space modeling proposed in this invention, a comparative experiment was set up to evaluate it from two aspects: fusion quality and key structural design.

[0176] First, by comparing with existing representative methods, the comprehensive fusion capability of the method of this invention in terms of spatial detail restoration and spectral preservation is evaluated. Second, through ablation analysis of the high-order state modeling mechanism of key modules, the actual contribution of the structural design proposed in this invention to the fusion performance is further verified. All experiments were conducted under uniform training conditions and quantitatively evaluated using mainstream evaluation metrics (PSNR, SSIM, SAM, ERGAS, QNR, etc.).

[0177] Experiment 1: Performance comparison with mainstream methods:

[0178] To verify the effectiveness of the method of this invention in remote sensing image fusion tasks, it is compared and analyzed with current mainstream deep learning fusion methods. The comparison methods include convolutional neural network models (such as PanNet and MSDCNN), Transformer architectures (such as INNformer and Panformer), and state-space modeling methods (such as PanMamba). Experiments were conducted on two typical remote sensing datasets, GF2 and WV3, and evaluation metrics included mainstream indicators such as peak signal-to-noise ratio (PSNR), spectral angle mapping (SAM), and abnormally normalized synthesis error (ERGAS).

[0179] The comparison results are shown in Tables 1 and 2. The method of this invention outperforms existing methods in multiple metrics. Specifically, the PSNR on the GF2 and WV3 datasets is improved by 0.19 dB and 0.22 dB respectively compared to the suboptimal method, significantly improving the spatial reconstruction quality of the images. Furthermore, in terms of spectral consistency metrics (such as SAM and ERGAS), the method of this invention achieves the best results on all test samples, further verifying its accuracy and stability in preserving spectral structure.

[0180] Table 1. Comparison of fusion capabilities between the method of this invention and the mainstream method Gao-fen 2.

[0181]

[0182]

[0183] Table 2 Comparison of fusion capabilities of the present invention method and mainstream methods on World-View III

[0184]

[0185] Figures 4 to 9The visualization of the proposed method on typical samples of the World-View II dataset is presented, including ground truth images, degraded low-resolution input images, fused images of each comparison method, and error maps (mean squared error, MSE) between the predicted and reference images. The images show that the image generated by the proposed method is the clearest in terms of detail restoration, and the dark areas constitute the largest proportion of the error map, indicating higher accuracy in preserving spatial structure and spectral features.

[0186] In summary, this experiment demonstrates that the method proposed in this invention exhibits superior performance under multiple evaluation conditions, possessing excellent spatial detail restoration capabilities and spectral consistency, and is suitable for image fusion tasks in various remote sensing scenarios.

[0187] Experiment 2: Ablation Validation of the Higher-Order State-Space Modeling Module:

[0188] To further verify the effectiveness of the high-order state-space modeling module proposed in this invention, related ablation experiments were conducted. These experiments primarily evaluated the impact of each key module (including ordinary convolution, Vanilla Mamba, bidirectional VMamba, and high-order PHOM) on image fusion quality. The evaluation metrics used included Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Spectral Angle Mapping (SAM), and Relative Unnormalized Synthesis Error (ERGAS).

[0189] This experiment was conducted on three typical remote sensing datasets: WorldView-II (WV2), Gaofen-2 (GF2), and WorldView-III (WV3). When constructing the comparison groups, the traditional ordinary convolutional module was used as the baseline, and then successively replaced with Vanilla Mamba (standard state-space modeling), VMamba (an improved structure incorporating a multi-directional scanning mechanism), and the PHoM module proposed in this invention (emphasizing a high-order state path design).

[0190] Table 3 Ablation verification of the high-order state-space modeling module of the present invention.

[0191]

[0192] The experimental results are shown in Table 3. On all three datasets, the algorithm employing the state-space model significantly outperformed the baseline method using traditional convolution, demonstrating that introducing a state-space modeling mechanism can effectively improve image fusion quality. In particular, the improvement of the VMamba module compared to Vanilla Mamba on the GF2 and WV3 datasets verifies the advantages of the multi-directional scanning mechanism in spatial modeling. Furthermore, the PHOM module proposed in this invention achieved the best PSNR and ERGAS metrics on all three datasets, fully demonstrating the significant performance advantages of its high-order structure design in complex image modeling.

[0193] The above results verify the superior comprehensive performance of the present invention in maintaining spectral consistency and enhancing spatial detail representation. They also demonstrate that the proposed high-order state path structure has strong practical engineering application value in remote sensing image fusion tasks.

[0194] In summary, the method described in this invention constructs a novel neural network structure. By introducing a high-order structured state-space model, this method employs a stage-by-stage recursive modeling strategy, effectively enhancing the spatial-spectral information interaction capability between multispectral and panchromatic images, and possessing linear computational complexity and global modeling capabilities. Simultaneously, a cross-modal extension structure is proposed, further improving the high-order interactive representation capability between multimodal data (such as multispectral and panchromatic images). Compared to existing convolutional neural networks, Transformer structures, and traditional state-space methods, this invention significantly improves spatial resolution and fusion efficiency while maintaining image spectral consistency, exhibiting strong adaptability and versatility. Experimental results show that this invention can be widely applied to various remote sensing satellite image fusion tasks, demonstrating good practical application value.

[0195] Furthermore, this invention also provides a pan-color sharpening method system based on high-order state space modeling, which uses the aforementioned pan-color sharpening method based on high-order state space modeling, including:

[0196] The image preprocessing module is used to acquire pairs of remote sensing images with different resolutions, including low-resolution multispectral images and high-resolution panchromatic images, and to upsample the multispectral images to make their resolution consistent with that of the panchromatic images, thus obtaining the upsampled multispectral images.

[0197] The feature extraction module is used to extract features from the upsampled multispectral image I. MS Compared with the original panchromatic image I PAN Shallow feature extraction is performed to obtain its initial feature representation, denoted as F. MS With F PAN ;

[0198] A high-order state space modeling module construction module is used to construct a high-order state space modeling module. The extracted features are input into the high-order state space modeling module, which uses structured state space units to jointly model spatial and channel information.

[0199] The modal-level dynamic fusion and high-order structure modeling module is used to enhance multispectral image I... MS With panchromatic image I PAN Cross-modal information interaction between them introduces a cross-modal fusion path, including a channel exchange module and a cross-modal fusion module, to achieve modal-level dynamic fusion and high-order structure modeling;

[0200] The high-resolution multispectral image output module is used to input the fused features into the reconstruction module and output a high-resolution multispectral image.

[0201] In some embodiments, the present invention provides a computer-readable storage medium storing computer instructions that are executed by a processor as described in any of the above embodiments, a full-color sharpening method based on high-order state-space modeling.

[0202] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (not an exhaustive list) of readable storage media may include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0203] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0204] Embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in a pan-color sharpening method based on high-order state space modeling according to various embodiments of the present invention, as described in the "Exemplary Methods" section above.

[0205] The steps of the method of the present invention are not limited to the specific order described above, unless otherwise specifically stated. Furthermore, in some embodiments, the invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the method according to the invention. Therefore, the invention also covers recording media storing programs for performing the method according to the invention.

[0206] Although the invention has been described with reference to preferred embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, the technical features mentioned in the various embodiments can be combined in any manner as long as there is no structural conflict. The invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A panchromatic sharpening method based on high-order state space modeling, characterized in that, Includes the following steps: Step 1: Acquire pairs of remote sensing images with different resolutions, including low-resolution multispectral images and high-resolution panchromatic images. Upsample the multispectral images to make their resolution consistent with that of the panchromatic images, and obtain the upsampled multispectral images. Step 2: Process the upsampled multispectral images separately. Compared with the original panchromatic image Shallow feature extraction is performed to obtain its initial feature representation, denoted as follows: F MS and F PAN ; Step 3: Construct a high-order state space modeling module. Input the extracted features into the high-order state space modeling module. The high-order state space modeling module uses structured state space units to jointly model spatial and channel information. Step 4: To enhance multispectral images With panchromatic image Cross-modal information interaction between them introduces a cross-modal fusion path, including a channel exchange module and a cross-modal fusion module, to achieve modal-level dynamic fusion and high-order structure modeling; Step 5: Input the fused features into the reconstruction module and output a high-resolution multispectral image; Step 4 specifically includes: Step 4.1, Channel Exchange Cross-fusion: Before entering the fusion stage, intermediate feature tensors from the multispectral path and the panchromatic path are extracted respectively. The channel exchange module is used to perform cross-fusion in the channel dimension. The specific operation is as follows: ， , By selecting some channels from the PAN features to replace the corresponding channels in the MS features, or by selecting some channels from the MS features to replace the corresponding channels in the PAN features, the initial injection and interaction of complementary structural information can be achieved. Step 4.2, Cross-modal fusion module: To further model cross-modal high-order representation information, a guided enhancement mechanism is adopted, which guides the fusion of panchromatic channel features by information-rich multispectral features. Specifically, this includes: ， The cross-fused features are fed into the cross-modal fusion module for further modeling. The MS channel features, which have richer information content, are used as guides. After linear mapping and SiLU activation, they are subjected to element-wise Hadamard product operation with the PAN channel features to realize the fusion guidance mechanism and enhance the collaborative expression between modalities.

2. The panchromatic sharpening method based on high-order state space modeling according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Acquire high-resolution multispectral images and their corresponding panchromatic images, and perform cropping operations to construct an image dataset; the cropping operation obtains pairs of remote sensing images with different resolutions, including low-resolution multispectral images. and high-resolution panchromatic images ;in, H The height of the high-resolution panchromatic image is represented by h, the width of the high-resolution panchromatic image is represented by w, and the number of channels of the high-resolution panchromatic image is represented by c. The corresponding height of the low-resolution multispectral image is h=H / 4, the width of the low-resolution multispectral image is w=W / 4, and the number of channels of the low-resolution multispectral image is represented by c. Step 1.2: Upsample the low-resolution multispectral image to match the resolution of the high-resolution panchromatic image using interpolation methods to obtain the upsampled multispectral image. .

3. The panchromatic sharpening method based on high-order state space modeling according to claim 2, characterized in that, Step 2 specifically includes: Step 2.1: Process the input image using a two-dimensional convolution operator with learnable weight parameters. The kernel size is [missing value]. With a stride of 1 and padding of 1 to maintain the output size, the result of the convolution operation is represented as: in, This represents the convolution operation. For convolution kernel weights, For bias terms; Step 2.2: To enhance the numerical stability of the model under different image samples, the instance normalization module is used to normalize the convolutional features. The calculation formula is as follows: in, and These represent the mean and variance for each channel. and To learnable affine transformation parameters, To prevent division by zero of small constants; Step 2.3: Select SiLU as the nonlinear activation function to enhance the network's representational power. The definition of SiLU is: in, Represents the Sigmoid function; For multispectral images With panchromatic image The shallow feature extraction process is represented as follows: 。 4. The panchromatic sharpening method based on high-order state space modeling according to claim 3, characterized in that, Step 3 specifically includes: Step 3.1: Design three independent state evolution paths, each corresponding to a feature subspace of different scales or semantic dimensions, with channel dimensions of D / 4, D / 2 and D, respectively, to express the spatial context modeling capability from shallow to deep and from local to global. Let the input feature tensor be ,in Where D is the number of spatial positions after flattening, and D is the number of channels. The specific modeling process is as follows: Step 3.1.1, Channel Upscaling and Partitioning: First, analyze the input features... Perform a linear transformation to increase its dimension to 2D, then divide it into three groups of sub-features proportionally: in, , , These represent the feature subspaces corresponding to the three sub-paths; Step 3.1.2, Three-way state evolution modeling: Each path uses an independent state-space evolution model, combining depthwise convolution and the SS2D operator to model its sequential state evolution process: in, Indicates the first i The depth of the branches can be separable from the convolutional operations for local feature enhancement; This represents a two-dimensional state-space modeling module used to capture long-distance dependencies between spatial locations; SS2D introduces a state-space-based dynamic modeling mechanism that captures long-range dependencies and temporal context information in remote sensing image sequences through the temporal evolution of latent states. This mechanism can be viewed as a modeling framework based on linear dynamic systems, and its continuous-time form is expressed as follows: ， in, Indicates the current input signal. This indicates the corresponding output response. The hidden state vector. Here is the state transition matrix. For the input mapping matrix, To output the mapping matrix; To meet the demands of computers processing discrete time series data, a zero-order hold strategy is introduced to discretize the above continuous model, resulting in a discrete state transition matrix and an input mapping matrix: ， in, For a fixed time step, The identity matrix is ​​represented; subsequently, sequence modeling is performed based on the discrete state-space model: This modeling process is equivalent to a structured convolution process, and its kernel function is defined as: in This represents a one-dimensional convolution operation. L Indicates the length of the input sequence.

5. The panchromatic sharpening method based on high-order state space modeling according to claim 4, characterized in that, Step 5 specifically includes: Step 5.1: Input the fused features into the image reconstruction module, which contains a 3×3 convolutional layer, to generate the output high-resolution multispectral image HrMS. Step 5.2: Introduce upsampled image MS through skip connections to improve detail restoration capabilities: 。 6. The panchromatic sharpening method based on high-order state space modeling according to claim 5, characterized in that, It also includes: Step 6, training the entire network structure through supervised learning, with the loss function including spatial error, structural similarity, and spectral consistency terms to achieve dual optimization of image sharpness and spectral fidelity; Step 6 specifically includes: Step 6.1, Spatial Domain Error Term: This term measures the overall error between the reconstructed image and the reference image in pixel space, using the mean absolute error as the metric. in, Output images to the network. For the corresponding high-resolution multispectral reference image; Step 6.2, Spectral Consistency Term: To maintain the spectral consistency of the fused image, a spectral angle mapping or scale-free relative global error metric is introduced as an additional regularization term to control spectral angle deviation and global error, using SAM: in, Output images to the network. For the corresponding high-resolution multispectral reference image; Step 6.3: The final combination of multinomial loss functions is a weighted sum, defined as follows: in, and This is a hyperparameter used to balance the impact weights of various losses.

7. A full-color sharpening system based on high-order state space modeling, characterized in that, The method using any one of claims 1 to 6 includes: The image preprocessing module is used to acquire pairs of remote sensing images with different resolutions, including low-resolution multispectral images and high-resolution panchromatic images, and to upsample the multispectral images to make their resolution consistent with that of the panchromatic images, thus obtaining the upsampled multispectral images. The feature extraction module is used to process the upsampled multispectral images. Compared with the original panchromatic image Shallow feature extraction is performed to obtain its initial feature representation, denoted as follows: F MS and F PAN ; A high-order state space modeling module construction module is used to construct a high-order state space modeling module. The extracted features are input into the high-order state space modeling module, which uses structured state space units to jointly model spatial and channel information. The modal-level dynamic fusion and high-order structure modeling module is used to enhance multispectral images. With panchromatic image Cross-modal information interaction between them introduces a cross-modal fusion path, including a channel exchange module and a cross-modal fusion module, to achieve modal-level dynamic fusion and high-order structure modeling; The high-resolution multispectral image output module is used to input the fused features into the reconstruction module and output a high-resolution multispectral image.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are executed by a processor according to any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product stores computer instructions that are executed by a processor using the method as described in any one of claims 1 to 6.

Images