Panchromatic sharpening method based on high-order state space modeling

Abstract

The invention discloses a panchromatic sharpening method based on high-order state space modeling, which effectively improves the space detail reduction and spectrum fidelity capability of a remote sensing image by introducing a hierarchical state modeling mechanism and a multi-stage joint learning framework, and has good performance stability, modeling capability and practical application value. The method comprises the following steps: constructing a deep neural network architecture for multispectral and panchromatic image fusion, and firstly, carrying out up-sampling on a low-resolution multispectral image to match the spatial dimension of a panchromatic image; then space and spectrum multi-scale features are extracted through a multi-stage feature extraction module; introducing a fusion module based on high-order structured state space modeling, wherein the module constructs a state updating process with long-range dependence modeling capability; information integration and image restoration are realized through a cross-stage residual connection and feature reconstruction module; semantic alignment and high-order interaction between multispectral and panchromatic modals are realized through a cross-modal fusion module, and the generalization performance of the model in a complex scene is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing and image fusion, and particularly relates to a panchromatic sharpening method based on high-order state space modeling. BACKGROUND

[0002] With the rapid development of remote sensing imaging technology, remote sensing satellites are widely used in resource investigation, urban planning, environmental monitoring and other fields. Multispectral (MS) images have rich spectral information but low spatial resolution, while panchromatic (PAN) images have high spatial resolution but lack spectral resolution. In order to obtain high spatial and high spectral information at the same time, image fusion technology, especially panchromatic sharpening, has become a research hotspot.

[0003] Traditional image fusion methods such as principal component analysis (PCA), intensity-hue-saturation transformation (IHS), Brovey transformation and generalized Gaussian filtering, although have high computational efficiency, but usually have problems of spectral distortion or lack of spatial details. In recent years, with the development of deep learning technology, convolutional neural network (CNN) and Transformer architecture have been widely introduced into the field of image fusion, which has significantly improved the quality of the fused image. However, the CNN structure is limited by the receptive field and is difficult to model long-range dependencies; while the Transformer has global modeling ability, but its high computational overhead poses a challenge to resource-constrained scenarios.

[0004] In addition, as a modeling framework with continuous modeling capability and linear time complexity, state space model (SSM) has shown broad prospects in image modeling, time series analysis and other directions. However, existing SSM-based methods mostly stay at first-order or simplified structure, and have not effectively tapped the potential of high-order spatial-spectral interaction in image fusion tasks.

[0005] Therefore, there is an urgent need for a new remote sensing image fusion method that can maintain spectral consistency while considering spatial detail enhancement and computational efficiency, in order to meet the dual demands of fusion image quality and model deployability in practical applications. SUMMARY

[0006] In view of the problems of limited spatial feature modeling capability, high computational complexity and insufficient multi-modal interaction modeling in the above remote sensing image fusion method, the application provides a panchromatic sharpening method based on high-order state space modeling, which is used to solve the problems of insufficient spatial-spectral information interaction, limited long-range dependent modeling and low model deployment efficiency in multispectral image and panchromatic image fusion. The application effectively improves the spatial detail and spectral preservation capability of the fused image by introducing a high-order state space modeling mechanism and a recursive stage structure, while having good engineering deployability, meeting the dual needs of precision and efficiency of remote sensing image analysis.

[0007] To achieve the above object, the technical scheme adopted by the application is as follows:

[0008] In a first aspect, the application provides a panchromatic sharpening method based on high-order state space modeling, comprising the following steps:

[0009] Step 1, obtaining a pair of remote sensing images with different resolutions, including a low-resolution multispectral image and a high-resolution panchromatic image, and performing up-sampling processing on the multispectral image to make its resolution consistent with that of the panchromatic image, to obtain an up-sampled multispectral image;

[0010] Step 2, performing shallow feature extraction processing on the up-sampled multispectral image I MS and the original panchromatic image I PAN to obtain their initial feature representations, respectively denoted as F MS and F PAN ;

[0011] Step 3, constructing a high-order state space modeling module, inputting the extracted features into the high-order state space modeling module, and using a structured state space unit to jointly model the spatial and channel information in the high-order state space modeling module;

[0012] Step 4, to strengthen the cross-modal information interaction between the multispectral image I MS and the panchromatic image I PAN , introducing a cross-modal fusion path, including a channel exchange module and a cross-modal fusion module, to realize modal-level dynamic fusion and high-order structure modeling;

[0013] Step 5, inputting the fused features into a reconstruction module to output a high-resolution multispectral image.

[0014] In an embodiment of the application, the step 1 specifically comprises:

[0015] Step 1.1, obtain high-resolution multispectral images and corresponding panchromatic images and perform cropping operation to construct an image dataset; the cropping operation obtains remote sensing image pairs with different resolutions, including low-resolution multispectral images and high-resolution panchromatic images and high-resolution panchromatic images wherein H represents the height of the high-resolution panchromatic image, W represents the width of the high-resolution panchromatic image, and c represents the channel number of the high-resolution panchromatic image; the corresponding h is the height of the low-resolution multispectral image, which is set as h = H / 4 here, w is the width of the low-resolution multispectral image, w = W / 4, and C represents the channel number of the low-resolution multispectral image;

[0016] Step 1.2, upsample the low-resolution multispectral image to the resolution consistent with that of the high-resolution panchromatic image by an interpolation method to obtain an upsampled multispectral image

[0017] In an embodiment of the present application, the step 2 specifically comprises:

[0018] Step 2.1, a two-dimensional convolution operator with learnable weight parameters is used to process the input image, the convolution kernel size is 3x3, the step is 1, and the padding is 1 to keep the output size unchanged, and the convolution operation result is represented as:

[0019] F (1) =W conv *I+b conv

[0020] wherein * represents the convolution operation, W conv is the convolution kernel weight, and b conv is the bias term;

[0021] Step 2.2, in order to enhance the numerical stability of the model under different image samples, an instance normalization module is used to normalize the convolution features, and the calculation formula is as follows:

[0022]

[0023] wherein μ and σ 2 are the mean and variance of each channel, γ and β are learnable affine transformation parameters, and ∈ is a small constant to prevent division by zero;

[0024] Step 2.3, SiLU is selected as a nonlinear activation function to enhance the representation ability of the network, and the definition of SiLU is as follows:

[0025] F (3) =SiLU(F (2) )=F (2) ·σ(F (2) )

[0026] wherein, σ(·) denotes a Sigmoid function;

[0027] For multi-spectral image I MS with panchromatic image I PAN The shallow feature extraction process is represented as:

[0028] F MS = SiLU(InstanceNorm(Conv 3×3 (I MS ))

[0029] F PAN = SiLU(InstanceNorm(Conv 3×3 (I PAN ))).

[0030] In an embodiment of the present application, the step 3 specifically comprises:

[0031] Step 3.1, design three independent state evolution paths, respectively corresponding to three different scale or semantic dimension feature subspaces, whose channel dimensions are D / 4, D / 2 and D respectively, for expressing the spatial context modeling ability from shallow to deep, from local to global;

[0032] Let the input feature tensor be wherein, L = H x W is the number of flattened spatial positions, and D is the number of channels, and the specific modeling process is as follows:

[0033] Step 3.1.1, channel dimension increase and division: first, linearly transform the input feature X to increase its dimension to 2D, and then divide it into three groups of sub-features in proportion:

[0034]

[0035] wherein, respectively represent the feature subspaces corresponding to the three sub-paths;

[0036] Step 3.1.2, three-path state evolution modeling: each path adopts an independent state space evolution model, combining deep convolution and SS2D operator to model its sequence state evolution process:

[0037]

[0038] wherein, DWConv i (·) denotes the depth separable convolution operation of the i-th branch, which is used for local feature enhancement; SS2D i (·) denotes a two-dimensional state space modeling module, which is used to capture long-distance dependencies between spatial positions;

[0039] SS2D introduces a state space based dynamic modeling mechanism to capture long-range dependencies and temporal context information in remote sensing image sequences through the temporal evolution process of hidden states.

[0040] h'(t) = Ah(t) + Bx(t), y(t) = Ch'(t)

[0041] wherein, represents the current input signal, represents the corresponding output response, is a hidden state vector, is a state transition matrix, is an input mapping matrix, is an output mapping matrix;

[0042] In order to adapt to the requirement of computer processing discrete time sequence data, a zero-order hold strategy is introduced to discretize the above continuous model to obtain a discrete state transition matrix and an input mapping matrix:

[0043]

[0044] wherein, Delta is a fixed time step, and I represents a unit matrix; subsequently, sequence modeling is performed based on the discrete state space model:

[0045]

[0046] y t = Ch t

[0047] The modeling process is equivalent to a structured convolution process, and the kernel function is defined as:

[0048]

[0049] wherein, * represents a one-dimensional convolution operation, and L represents the length of the input sequence.

[0050] In an embodiment of the present application, the step 4 specifically comprises:

[0051] Step 4.1, channel exchange cross fusion: before entering the fusion stage, the intermediate feature tensors from the multi-spectral path and the panchromatic path are extracted respectively, and a channel exchange module is used to cross-fuse in the channel dimension, and the specific operation is as follows:

[0052]

[0053] Some channels in the PAN features are replaced by corresponding channels in the MS features, or some channels in the MS features are replaced by corresponding channels in the PAN features, to achieve preliminary injection and interaction of complementary structural information;

[0054] Step 4.2, cross-modal fusion module: in order to further model the cross-modal high-order expression information, a guided enhancement mechanism is adopted, in which the multispectral features with rich information content guide the fusion of the panchromatic channel features, and the specific steps include:

[0055]

[0056] The cross-fused features are input into the cross-modal fusion module for further modeling, the MS channel features with richer information content are used as the guide, linear mapping and SiLU activation are performed on the MS channel features, and then the panchromatic channel features are subjected to element-wise Hadamard product operation, so that the fusion guide mechanism is realized, and the collaborative expression between the modes is enhanced.

[0057] In an embodiment of the present application, the step 5 specifically includes:

[0058] Step 5.1, the fused features are input into an image reconstruction module, which includes a 3*3 convolution layer, for generating an output high-resolution multispectral image HrMS:

[0059]

[0060] Step 5.2, an up-sampled image MS is introduced through a skip connection, so as to improve the detail restoration capability:

[0061]

[0062] In an embodiment of the present application, it further includes: step 6, the entire network structure is trained through supervised learning, and a loss function includes a spatial error term, a structural similarity term and a spectral consistency term, so as to realize dual optimization of image clarity and spectral fidelity; the step 6 specifically includes:

[0063] Step 6.1, spatial domain error term: this term is used to measure the overall error of the reconstructed image and the reference image in the pixel space, and the mean absolute error is used as the metric:

[0064]

[0065] Wherein, I HrMS is the network output image, and I GT is the corresponding high-resolution multispectral reference image;

[0066] Step 6.2, spectral consistency term: to maintain the spectral consistency of the fused image, a spectral angle mapping or a scale-invariant relative global error metric indicator is introduced as an additional regularization term for controlling the spectral angle deviation and the global error, using SAM:

[0067]

[0068] where I HrMS is the network output image, I GT is the corresponding high-resolution multispectral reference image;

[0069] Step 6.3, the final multi-term loss function combination is in the form of a weighted sum, defined as follows:

[0070]

[0071] where λ1 and λ2 are hyperparameters for balancing the influence weight of each loss term.

[0072] In a second aspect, the present application provides a panchromatic sharpening system based on high-order state space modeling, which uses the method, comprising:

[0073] An image preprocessing module is used to obtain a pair of remote sensing images with different resolutions, including a low-resolution multispectral image and a high-resolution panchromatic image, and to perform up-sampling processing on the multispectral image to make its resolution consistent with that of the panchromatic image, thereby obtaining an up-sampled multispectral image;

[0074] A feature extraction module is used to respectively perform shallow feature extraction processing on the up-sampled multispectral image I MS and the original panchromatic image I PAN to obtain their initial feature representations, denoted as F MS and F PAN respectively.

[0075] A high-order state space modeling module is used to construct a high-order state space modeling module, and the extracted features are input into the high-order state space modeling module, which uses a structured state space unit to jointly model spatial and channel information.

[0076] A modal-level dynamic fusion and high-order structure modeling module is used to strengthen the cross-modal information interaction between the multispectral image I MS and the panchromatic image I PAN , and a cross-modal fusion path is introduced, including a channel exchange module and a cross-modal fusion module, to realize modal-level dynamic fusion and high-order structure modeling.

[0077] A high-resolution multispectral image output module is used to input the fused features into a reconstruction module to output a high-resolution multispectral image.

[0078] In a third aspect, the present application provides a computer readable storage medium storing computer instructions for causing a processor to perform the method described above.

[0079] In a fourth aspect, the present application provides a computer program product storing computer instructions for causing a processor to perform the method described above.

[0080] The present application has the following beneficial effects:

[0081] The present application provides a panchromatic sharpening method based on high-order state space modeling, which effectively improves the spatial detail restoration and spectral fidelity of remote sensing images by introducing a hierarchical state modeling mechanism and a multi-stage joint learning framework, and has good performance stability, modeling ability and practical application value. The method constructs a deep neural network architecture for multispectral and panchromatic image fusion. First, the low-resolution multispectral image is upsampled to match the spatial dimension of the panchromatic image; then the spatial and spectral multi-scale features are extracted through a multi-stage feature extraction module; then a fusion module based on high-order structured state space modeling is introduced, which adopts one-dimensional convolution, gating mechanism and selective scanning mechanism to construct a state update process with long-range dependency modeling capability; further, information integration and image restoration are realized through cross-stage residual connection and feature reconstruction module; in addition, a cross-modal fusion module (C-PHoM) is proposed to realize semantic alignment and high-order interaction between multispectral and panchromatic modalities, and enhance the generalization performance of the model in complex scenes. Thus, the fusion quality of the present application is higher: it can effectively enhance the spatial details and maintain the spectral consistency; the modeling capability is stronger: based on the high-order state modeling mechanism, the long-range spatial dependency in the image can be mined; the calculation efficiency is more optimal: the state space structure with linear complexity is suitable for large-scale remote sensing image fusion tasks; the adaptability is more extensive: it supports the fusion of various remote sensing images (such as WorldView-II, GF2, WV3, etc.), and has good engineering application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0082] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the drawings.

[0083] Figure 1 The present application provides a method for the whole flowchart.

[0084] Figure 2Mamba module framework diagram used for the method of the present application.

[0085] Figure 3 High-order state space modeling module framework diagram used for the method of the present application.

[0086] Figure 4 Subjective experiment effect diagram on WorldView-II dataset using the method of the present application and traditional panchromatic sharpening method, deep learning panchromatic sharpening method;

[0087] Figure 5 Error experiment effect diagram on WorldView-II dataset using the method of the present application and traditional panchromatic sharpening method, deep learning panchromatic sharpening method;

[0088] Figure 6 Subjective experiment effect diagram on Gaofen-2 dataset using the method of the present application and traditional panchromatic sharpening method, deep learning panchromatic sharpening method;

[0089] Figure 7 Error experiment effect diagram on Gaofen-2 dataset using the method of the present application and traditional panchromatic sharpening method, deep learning panchromatic sharpening method;

[0090] Figure 8 Subjective experiment effect diagram on WorldView-III dataset using the method of the present application and traditional panchromatic sharpening method, deep learning panchromatic sharpening method;

[0091] Figure 9 Error experiment effect diagram on WorldView-III dataset using the method of the present application and traditional panchromatic sharpening method, deep learning panchromatic sharpening method. DETAILED DESCRIPTION

[0092] The embodiments of the technical solutions of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0093] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above description of drawings are intended to cover non-exclusive inclusion.

[0094] In the description of the embodiments of the present application, the technical terms "first", "second" and the like are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0095] Reference herein to "embodiments" means that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least some embodiments of the application. The occurrence of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor does it necessarily refer to a separate or alternative embodiment, which is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with each other.

[0096] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0097] As shown in Figures 1 to 3 The present application provides a high-order state space modeling-based panchromatic sharpening method, comprising the following steps:

[0098] Step 1, input image preparation, including data acquisition and preprocessing; the step 1 specifically includes:

[0099] Step 1.1, obtaining high-resolution multispectral images and corresponding panchromatic images and performing cropping operation, thereby constructing an image dataset; the cropping operation obtains remote sensing image pairs with different resolutions, including low-resolution multispectral images (LRMS) and high-resolution panchromatic images (PAN) Wherein, H represents the height of the high-resolution panchromatic image, W represents the width of the high-resolution panchromatic image, and c represents the channel number of the high-resolution panchromatic image, which is 1 by default; the corresponding h is the height of the low-resolution multispectral image, which is set to h=H / 4 here, w is the width of the low-resolution multispectral image, w=W / 4, and C represents the channel number of the low-resolution multispectral image;

[0100] Step 1.2, up-sampling the low-resolution multispectral image to the same resolution as the high-resolution panchromatic image by an interpolation method (such as bicubic interpolation), to obtain an up-sampled multispectral image

[0101] Step 2, up-sampling the multispectral image I MS and the original panchromatic image IPAN A shallow feature extraction process is performed to obtain its initial feature representation, denoted as F MS and F PAN , respectively.

[0102] Step 2.1, a two-dimensional convolution operator with learnable weight parameters is used to process the input image, with a convolution kernel size of 3x3, a step size of 1, and padding of 1 to maintain the output size unchanged. The convolution operation result is represented as:

[0103] F (1) = W conv *I+b conv

[0104] where * represents the convolution operation, W conv is the convolution kernel weight, and b conv is the bias term.

[0105] Step 2.2, to enhance the numerical stability of the model under different image samples, an instance normalization (Instance Normalization) module is used to normalize the convolution features, and its calculation formula is as follows:

[0106]

[0107] where μ and σ 2 are the mean and variance of each channel, γ and β are learnable affine transformation parameters, and ∈ is a small constant to prevent division by zero.

[0108] Step 2.3, SiLU (Sigmoid Linear Unit) is selected as the nonlinear activation function to enhance the representation ability of the network, and the definition of SiLU is as follows:

[0109] F (3) = SiLU(F (2) ) = F (2) ·σ(F (2) )

[0110] where σ(·) represents the Sigmoid function.

[0111] In summary, for the multispectral image I MS and the panchromatic image I PAN , the shallow feature extraction process is represented as:

[0112] F MS = SiLU(InstanceNorm(Conv 3×3 (I MS )))

[0113] F PAN= SiLU (InstanceNorm (Conv 3×3 (I PAN )))

[0114] Further, the feature extraction module can stack multiple such operation units to form a shallow convolutional encoder with a residual structure. The role is to provide stable, efficient and information-accurate modal feature representation, providing high-quality input for subsequent state space modeling and modal fusion.

[0115] Step 3, construct a high-order state space modeling module, the core of the present application is to construct a high-order state space modeling module with multi-path dynamic modeling capability, in order to more fully mine the complex spatial dependence relationship and semantic representation capability in multispectral image and panchromatic image; the step 3 specifically includes:

[0116] Step 3.1, design three independent state evolution paths, respectively corresponding to three different scale or semantic dimension feature subspaces, whose channel dimensions are D / 4, D / 2 and D, respectively, for expressing the spatial context modeling capability from shallow to deep, from local to global;

[0117] Let the input feature tensor be Where L=HxW is the number of flattened spatial positions, D is the number of channels, and the specific modeling process is as follows:

[0118] Step 3.1.1, channel dimension increase and division: first, linearly transform the input feature X to increase its dimension to 2D, and then divide it into three groups of sub-features in proportion:

[0119]

[0120] Wherein, Respectively represent the feature subspaces corresponding to the three sub-paths;

[0121] Step 3.1.2, three-path state evolution modeling: each path uses an independent state space evolution model, combining deep convolution and SS2D (Selective Scan 2D) operator to model its sequence state evolution process:

[0122]

[0123] Wherein, DWConv i (·) represents the depth separable convolution operation of the i-th branch, which is used for local feature enhancement; SS2D i (·) represents a two-dimensional state space modeling module, which is used to capture long-distance dependence between spatial positions;

[0124] SS2D introduces a state-space dynamic modeling (SSM) mechanism. Its core idea is to capture long-range dependencies and temporal context information in remote sensing image sequences through the temporal evolution of latent states. This mechanism can be viewed as a modeling framework based on linear dynamic systems, and its continuous-time form can be expressed as:

[0125] h′(t)=Ah(t)+Bx(t),y(t)=Ch′(t)

[0126] in, Indicates the current input signal. This indicates the corresponding output response. The hidden state vector. Here is the state transition matrix. For the input mapping matrix, To output the mapping matrix;

[0127] To meet the demands of computers processing discrete time series data, a zero-order hold (ZOH) strategy is introduced to discretize the aforementioned continuous model, resulting in a discrete state transition matrix and input mapping matrix:

[0128]

[0129] Where Δ is a fixed time step and I represents the identity matrix; subsequently, sequence modeling is performed based on the discrete state-space model:

[0130]

[0131] y t =Ch t

[0132] In its implementation, this modeling process is equivalent to a structured convolution process, and its kernel function is defined as:

[0133]

[0134] Where * represents a one-dimensional convolution operation and L represents the length of the input sequence; this modeling method not only has linear time complexity, but also enables long-range dependency modeling in the global scope, significantly enhancing the feature representation capability of multispectral images.

[0135] Step 4: To enhance the multispectral image I MS With panchromatic image I PANThe cross-modal information interaction between the multispectral and panchromatic channels is improved, and the spatial-spectral joint modeling capability is improved. The cross-modal fusion path is introduced into the network structure, including a channel swap module (ChannelSwap) and a cross-modal fusion module (C-PHoM), to realize modal-level dynamic fusion and high-order structure modeling. The step 4 specifically includes:

[0136] Step 4.1, channel exchange cross fusion (ChannelSwap): before entering the fusion stage, the intermediate feature tensors from the multispectral path and the panchromatic path are extracted, respectively, and the channel exchange module is used for cross fusion in the channel dimension, and the specific operation is as follows:

[0137]

[0138]

[0139] Some channels are selected from the PAN feature to replace the corresponding channels in the MS feature, and vice versa, to realize the preliminary injection and interaction of complementary structural information.

[0140] Step 4.2, cross-modal fusion module (C-PHoM): to further model the cross-modal high-order expression information, the module uses a guided enhancement mechanism, in which the multispectral feature with rich information guides the panchromatic channel feature fusion, and specifically includes:

[0141]

[0142] The cross-fused features are input into the cross-modal fusion module (C-PHoM) for further modeling, and the MS channel feature with richer information is used as a guide, which is linearly mapped, SiLU activated, and then element-wise Hadamard product is performed with the PAN channel feature, to realize the fusion guide mechanism and enhance the inter-modal collaborative expression.

[0143] Step 5, image reconstruction; the step 5 specifically includes:

[0144] Step 5.1, input the fusion features into the image reconstruction module, which includes a 3x3 convolution layer, for generating the output high-resolution multispectral image HrMS:

[0145]

[0146] Step 5.2, introduce the up-sampled image MS through the skip connection to improve the detail restoration capability.

[0147]

[0148] Step 6: Train the entire network structure through supervised learning. The loss function includes spatial error terms (such as mean squared error MSE), structural similarity (SSIM), and spectral consistency terms (such as SAM or ERGAS) to achieve dual optimization of image sharpness and spectral fidelity. Step 6 specifically includes:

[0149] Step 6.1, Spatial Domain Error Term: This term measures the overall error between the reconstructed image and the reference image in pixel space. Here, the metric used is Mean Absolute Error (MAE):

[0150]

[0151] Among them, I HrMS To output an image to the network, I GT For the corresponding high-resolution multispectral reference image;

[0152] Step 6.2, Spectral Consistency Term: To maintain the spectral consistency of the fused image, metrics such as Spectral Angle Mapper (SAM) or Scale-Free Relative Global Error (ERGAS) are introduced as additional regularization terms to control spectral angle deviation and global error. Here, SAM is used.

[0153]

[0154] Among them, I HrMS To output an image to the network, I GT For the corresponding high-resolution multispectral reference image;

[0155] Step 6.3: The final combination of multinomial loss functions is in the form of a weighted sum, defined as follows:

[0156]

[0157] λ1 and λ2 are hyperparameters used to balance the influence weights of various losses. This composite loss function can effectively guide the model to gradually improve the detail restoration ability and spectral preservation performance of the fused image during training, and achieve synergistic optimization of spatial and spectral information.

[0158] like Figure 1 The diagram shown is an overall flowchart of the method of this invention. The input data includes an upsampled low-resolution multispectral image and a high-resolution panchromatic image. The multispectral image is first upsampled to make its size consistent with the panchromatic image.

[0159] In the feature extraction stage (Stage 1), the panchromatic and up-sampled multispectral images are respectively subjected to 3x3 convolution and normalization for preliminary feature extraction, and are further mapped to a unified embedding dimension through a feature embedding module (Tokenization). Then, the network introduced multiple high-order Mamba blocks (Higher-Order Mamba Block) for deep spatial modeling of the two features. The focus of this stage is to capture the spatial structure and primary semantic features of the image.

[0160] In the feature fusion stage (Stage 2), the application uses a channel swap module (Channel Swap, CS) to exchange information between the multispectral and panchromatic features, and then sends the fused features to a deeper high-order Mamba block for joint modeling. In this stage, the application further introduces a residual connection structure and a SiLU activation function to enhance the feature expression capability while maintaining stable training. Finally, a 3x3 convolution and a skip connection are used to restore the high-resolution multispectral image (HrMS).

[0161] To further clarify the functions and structures of the key sub-modules in the fusion structure proposed by the application, the core constituent units are described in detail, including: Vanilla Mamba, Higher-Order Mamba Block, and C-PHoM.

[0162] As shown in Figure 2 , the basic Mamba structure (Vanilla Mamba) referred to by the application. This module is based on the idea of state space modeling, which maps a continuous-time dynamic system into a discrete update function. The core of the module includes the state update function h'(t) = Ah(t) + Bx(t), the output function y(t) = Ch'(t), and the structured parameter matrix (A, B, C, D). Multiple sub-modules such as linear mapping, embedding, normalization (LN), deep separable convolution (DWConv), and SiLU activation function are also introduced.

[0163] This structure has been verified to have good long-range dependency modeling capability in multiple modeling tasks, and is extended as the basic unit of the application.

[0164] As shown in Figure 3 , the Higher-Order Mamba Block structure diagram proposed by the application. Based on Vanilla Mamba, this module uses a multi-order state path design, i.e., multiple Selective Scan 2D (SS2D) state update channels are set up, each channel modeling spatial feature expression of different granularity.

[0165] The internal structure of the module is a three-layer serial state channel, which respectively models the spatial state of the channel dimension D / 4, D / 2 and D. Each path is composed of SS2D+DWConv+Linear+SiLU, and uses Hadamard product to fuse multi-order state representation at the output end. Finally, through the linear mapping layer, the output is unified to the original embedding dimension, realizing the joint modeling of complex spatial structure and long-range dependence.

[0166] Further, in order to strengthen the cross-modal information interaction between multispectral images and panchromatic images and improve the spatial-spectral joint modeling capability, the present application introduces a cross-modal fusion path in the network structure, including a channel exchange module (ChannelSwap) and a cross-modal fusion module C-PHoM, to realize modal-level dynamic fusion and high-order structure modeling.

[0167] The present application introduces a channel exchange cross-fusion (ChannelSwap) module in the network structure, which cross-fuses the channel dimensions of the intermediate feature tensors from the multispectral path and the panchromatic path before entering the fusion stage.

[0168] The present application introduces a C-PHoM (Cross-modal PHoM) module in the network structure, which is used to fuse and enhance MS features and PAN features at a specific stage, that is, by using the characteristics of rich MS feature information content, it is introduced into the linear transformation and SiLU activation through Hadamard product to dominate the fusion process.

[0169] The high-order module can flexibly stack multiple layers and support residual connection with the backbone features, which is used to enhance the deep modeling capability. The module can be widely used in visual tasks including image fusion, super-resolution, target detection, etc., and has good generalization and engineering implementability.

[0170] In summary, the present application proposes a panchromatic sharpening method based on high-order state space modeling, which effectively improves the spatial detail restoration and spectral fidelity of remote sensing images by introducing a hierarchical state modeling mechanism and a multi-stage joint learning framework, and has good performance stability, modeling ability and practical application value. The method constructs a deep neural network architecture for multispectral and panchromatic image fusion. First, the low-resolution multispectral image is upsampled to match the spatial dimension of the panchromatic image; then the spatial and spectral multi-scale features are extracted through the multi-stage feature extraction module; then the fusion module based on high-order structured state space modeling is introduced, which adopts one-dimensional convolution, gating mechanism and selective scanning mechanism to construct a state update process with long-range dependency modeling capability; further, through the cross-stage residual connection and feature reconstruction module, information integration and image restoration are realized; in addition, a cross-modal fusion module (C-PHoM) is proposed to realize semantic alignment and high-order interaction between multispectral and panchromatic modalities, and enhance the generalization performance of the model in complex scenes.

[0171] The effects of the present application can be further illustrated by the following simulation experiment.

[0172] 1. Simulation conditions and parameters

[0173] In the simulation experiment of the present application, three kinds of remote sensing image data sets, WorldView-II (WV2), Gaofen2 (GF2) and WorldView-III (WV3) are selected, and Wald protocol is used for simulated degradation to construct a no-reference fusion evaluation scene. The image spatial resolution ratio is 1:4, the input multispectral image size is 32x32x4, and the panchromatic image size is 128x128x1. The experiment is completed on a single NVIDIA3090Ti GPU, and is realized by using the PyTorch framework. The total number of iterations is 200, 500 and 500 respectively, the initial learning rate is set to 5x10 -4 , the optimizer is Adam, and the batch size is 4.

[0174] 2. Simulation content and result analysis

[0175] In the simulation experiment, in order to verify the effectiveness and advancement of the panchromatic sharpening method based on high-order state space modeling proposed in the present application, comparative experiments are set up, and the fusion quality and key structure design are evaluated from two aspects:

[0176] Firstly, the comprehensive fusion capability of the method in terms of spatial detail restoration and spectral preservation is evaluated by comparison with existing representative methods; secondly, the actual contribution of the structure design proposed in the application to the fusion performance is further verified through ablation analysis of the high-order state modeling mechanism of the key module. Experiments are carried out under uniform training conditions, and mainstream evaluation indexes (PSNR, SSIM, SAM, ERGAS, QNR, etc.) are used for quantitative evaluation.

[0177] Experiment 1: Fusion performance comparison with mainstream methods

[0178] To verify the effectiveness of the method in the remote sensing image fusion task, it is compared and analyzed with the current mainstream deep learning fusion method. The comparison methods include convolutional neural network models (such as PanNet, MSDCNN), Transformer architecture (such as INNformer, Panformer) and state space modeling methods (such as PanMamba, etc.). Experiments are carried out on two typical remote sensing datasets GF2 and WV3, and the evaluation indexes include peak signal-to-noise ratio (PSNR), spectral angle mapping (SAM), non-normalized synthetic error (ERGAS) and other mainstream indexes.

[0179] The comparison results are shown in Tables 1 and 2, and the method of the application is better than the existing methods in multiple indexes. Among them, the PSNR on the GF2 and WV3 datasets is improved by 0.19 dB and 0.22 dB respectively compared with the suboptimal method, which significantly improves the spatial reconstruction quality of the image. At the same time, in terms of spectral consistency indexes (such as SAM and ERGAS), the method of the application achieves the best results on all test samples, further verifying its accuracy and stability in preserving spectral structure.

[0180] Table 1 Fusion capability comparison of the method of the application and mainstream methods on Gao-fen 2

[0181]

[0182]

[0183] Table 2 Fusion capability comparison of the method of the application and mainstream methods on World-View III

[0184]

[0185] Figures 4 to 9The visualization fusion results of the method on typical samples of the World-View II data set are shown, including a ground truth, a degraded low-resolution input image, a fusion image of each comparison method, and an error map (mean square error, MSE) between the predicted image and the reference image. It can be observed from the figure that the image generated by the method is the clearest in detail recovery, and the dark area accounts for the largest proportion in the error map, indicating that it has higher accuracy in maintaining spatial structure and spectral characteristics.

[0186] In summary, the experiment shows that the method proposed in the application has superior performance under multiple evaluation conditions, has good spatial detail restoration ability and spectral consistency, and is suitable for image fusion tasks in various remote sensing scenes.

[0187] Experiment 2: Ablation verification of the high-order state space modeling module

[0188] In order to further verify the effectiveness of the high-order state space modeling module proposed in the application, related ablation experiments are carried out. The experiment mainly evaluates the influence of each key module (including ordinary convolution, Vanilla Mamba, bidirectional VMamba, and high-order PHoM) on image fusion quality, and the evaluation indexes include peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), spectral angle mapping (SAM), and relative unnormalized synthesis error (ERGAS).

[0189] The experiment is carried out on three typical remote sensing data sets, WorldView-II (WV2), Gaofen-2 (GF2), and WorldView-III (WV3). In constructing the comparison group, the traditional ordinary convolution module is used as the baseline (Baseline), and is replaced with Vanilla Mamba (standard state space modeling), VMamba (improved structure with multi-directional scanning mechanism), and the PHoM module (high-order state path design) proposed in the application in turn.

[0190] Table 3 Ablation verification of the high-order state space modeling module of the method of the application

[0191]

[0192] The experimental results are shown in Table 3. On three data sets, the algorithm using the state space model is significantly better than the baseline method using traditional convolution, indicating that the introduction of the state modeling mechanism can effectively improve the image fusion quality. In particular, the improvement of the VMamba module compared with the Vanilla Mamba on the GF2 and WV3 data sets verifies the advantage of the multi-directional scanning mechanism in spatial modeling. And the PSNR and ERGAS indicators of the PHoM module proposed in the application reach the optimal on all three data sets, fully demonstrating that the high-order structure design has a significant performance advantage in complex image modeling.

[0193] The above results verify the superior comprehensive performance of the application in maintaining spectral consistency and enhancing spatial detail expression, and also show that the proposed high-order state path structure has strong practical engineering application value in remote sensing image fusion tasks.

[0194] In summary, the method described in the application constructs a new neural network structure, by introducing a high-order structured state space model, the method uses a stage-by-stage recursive modeling strategy, effectively enhancing the spatial-spectral information interaction capability between multi-spectral images and panchromatic images, with linear computational complexity and global modeling capability. At the same time, a cross-modal expansion structure is proposed, further improving the high-order interaction expression capability between multi-modal data (such as multi-spectral and panchromatic images). Compared with existing convolutional neural networks, Transformer structures and traditional state space methods, the application significantly improves spatial resolution and fusion efficiency while maintaining image spectral consistency, with strong adaptability and universality. Experimental results show that the application can be widely applied to the fusion tasks of various remote sensing satellite images, and has good practical application value.

[0195] In addition, the application also provides a panchromatic sharpening method system based on high-order state space modeling, which uses the panchromatic sharpening method based on high-order state space modeling, including:

[0196] An image preprocessing module is used to obtain a pair of remote sensing images with different resolutions, including a low-resolution multi-spectral image and a high-resolution panchromatic image, and to perform up-sampling processing on the multi-spectral image to make its resolution consistent with that of the panchromatic image, obtaining an up-sampled multi-spectral image;

[0197] A feature extraction module is used to perform shallow feature extraction processing on the up-sampled multi-spectral image I MS and the original panchromatic image I PAN to obtain their initial feature representations, respectively denoted as F MS and F PAN ;

[0198] The high-order state space modeling module constructs a high-order state space modeling module, inputs the extracted features into the high-order state space modeling module, and the high-order state space modeling module jointly models the spatial and channel information by using a structured state space unit;

[0199] The modal level dynamic fusion and high-order structure modeling module is used for strengthening the cross-modal information interaction between the multispectral image I MS and the panchromatic image I PAN , introducing a cross-modal fusion path, including a channel exchange module and a cross-modal fusion module, to realize modal level dynamic fusion and high-order structure modeling.

[0200] The high-resolution multispectral image output module is used for inputting the fused features into a reconstruction module and outputting a high-resolution multispectral image.

[0201] In some embodiments, the present application provides a computer readable storage medium storing computer instructions, which are executed by a processor to perform a panchromatic sharpening method based on high-order state space modeling according to any one of the above embodiments.

[0202] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of readable storage medium can include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0203] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing apparatus to produce a machine, so that the instructions executed by the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device for implementing the functions specified in one flow or multiple flows and / or one block or multiple blocks. Figure 1 The device for implementing the functions specified in one flow or multiple flows and / or one block or multiple blocks.

[0204] Embodiments of the present application can also be a computer program product which includes computer program instructions that, when run by a processor, cause the processor to perform steps of a high order state space modeling based panchromatic sharpening method according to various embodiments of the present application described in the above "Exemplary Methods" section of the specification.

[0205] The steps of the method of the present application are not limited to the order specifically described above, unless otherwise specifically stated. Furthermore, in some embodiments, the present application can also be implemented as programs recorded in recording media, which include machine readable instructions for implementing the method according to the present application. Thus, the present application also covers recording media storing programs for executing the method according to the present application.

[0206] Although the present application has been described with reference to preferred embodiments, various modifications can be made to it without departing from the scope of the present application, and equivalent parts can be substituted therefor. In particular, the technical features mentioned in each of the embodiments can be combined in any manner, provided that there is no structural conflict. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A panchromatic sharpening method based on high-order state space modeling, characterized in that, Includes the following steps: Step 1: Acquire pairs of remote sensing images with different resolutions, including low-resolution multispectral images and high-resolution panchromatic images. Upsample the multispectral images to make their resolution consistent with that of the panchromatic images, and obtain the upsampled multispectral images. Step 2: Process the upsampled multispectral images I MS Compared with the original panchromatic image I PAN Shallow feature extraction is performed to obtain its initial feature representation, denoted as F. MS With F PAN ; Step 3: Construct a high-order state space modeling module. Input the extracted features into the high-order state space modeling module. The high-order state space modeling module uses structured state space units to jointly model spatial and channel information. Step 4: To enhance the multispectral image I MS With panchromatic image I PAN Cross-modal information interaction between them introduces a cross-modal fusion path, including a channel exchange module and a cross-modal fusion module, to achieve modal-level dynamic fusion and high-order structure modeling; Step 5: Input the fused features into the reconstruction module and output a high-resolution multispectral image.

2. The panchromatic sharpening method based on high-order state space modeling according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Acquire high-resolution multispectral images and their corresponding panchromatic images, and perform cropping operations to construct an image dataset; the cropping operation obtains pairs of remote sensing images with different resolutions, including low-resolution multispectral images. and high-resolution panchromatic images Where H represents the height of the high-resolution panchromatic image, W represents the width of the high-resolution panchromatic image, and c represents the number of channels of the high-resolution panchromatic image; the corresponding h is the height of the low-resolution multispectral image, which is set here as h = H / 4, w is the width of the low-resolution multispectral image, w = W / 4, and C represents the number of channels of the low-resolution multispectral image. Step 1.2: Upsample the low-resolution multispectral image to match the resolution of the high-resolution panchromatic image using interpolation methods to obtain the upsampled multispectral image.

3. The panchromatic sharpening method based on high-order state space modeling according to claim 2, characterized in that, Step 2 specifically includes: Step 2.1: Process the input image using a two-dimensional convolution operator with learnable weight parameters. The kernel size is 3×3, the stride is 1, and the padding is 1 to maintain the output size. The result of the convolution operation is expressed as: F (1) =W conv *I+b conv Where * represents the convolution operation, W conv b represents the kernel weights. conv For bias terms; Step 2.2: To enhance the numerical stability of the model under different image samples, the instance normalization module is used to normalize the convolutional features. The calculation formula is as follows: Where μ and σ 2 denoted as the mean and variance for each channel, γ and β are learnable affine transformation parameters, and ∈ is a small constant to prevent division by zero; Step 2.3: Select SiLU as the nonlinear activation function to enhance the network's representational power. The definition of SiLU is: F (3) =SiLU(F (2) )=F (2) ·σ(F (2) ) Where σ(·) represents the Sigmoid function; For multispectral images I MS With panchromatic image I PAN The shallow feature extraction process is represented as follows: F MS =SiLU(InstanceNorm(Conv 3×3 (I MS ))) F PAN =SiLU(InstanceNorm(Conv 3×3 (I PAN )))。 4. The panchromatic sharpening method based on high-order state space modeling according to claim 3, characterized in that, Step 3 specifically includes: Step 3.1: Design three independent state evolution paths, each corresponding to a feature subspace of different scales or semantic dimensions, with channel dimensions of D / 4, D / 2 and D, respectively, to express the spatial context modeling capability from shallow to deep and from local to global. Let the input feature tensor be Where L = H × W is the number of spatial positions after flattening, and D is the number of channels. The specific modeling process is as follows: Step 3.1.1, Channel Dimension Upgrading and Partitioning: First, perform a linear transformation on the input feature X to increase its dimension to 2D, and then divide it into three groups of sub-features according to the proportions: in, These represent the feature subspaces corresponding to the three sub-paths; Step 3.1.2, Three-way state evolution modeling: Each path uses an independent state-space evolution model, combining depthwise convolution and the SS2D operator to model its sequential state evolution process: Among them, DWConv i (·) indicates a depthwise separable convolution operation in the i-th branch, used for local feature enhancement; SS2D i (·) represents a two-dimensional state-space modeling module, used to capture long-distance dependencies between spatial locations; SS2D introduces a state-space-based dynamic modeling mechanism that captures long-range dependencies and temporal context information in remote sensing image sequences through the temporal evolution of latent states. This mechanism can be viewed as a modeling framework based on linear dynamic systems, and its continuous-time form is expressed as follows: h′(t)=Ah(t)+Bx(t),y(t)=Ch′(t) in, Indicates the current input signal. This indicates the corresponding output response. The hidden state vector. Here is the state transition matrix. For the input mapping matrix, To output the mapping matrix; To meet the demands of computers processing discrete time series data, a zero-order hold strategy is introduced to discretize the above continuous model, resulting in a discrete state transition matrix and an input mapping matrix: Where Δ is a fixed time step and I represents the identity matrix; subsequently, sequence modeling is performed based on the discrete state-space model: the t =Ch t This modeling process is equivalent to a structured convolution process, and its kernel function is defined as: Where * represents a one-dimensional convolution operation, and L represents the length of the input sequence.

5. The panchromatic sharpening method based on high-order state space modeling according to claim 4, characterized in that, Step 4 specifically includes: Step 4.1, Channel Exchange Cross-fusion: Before entering the fusion stage, intermediate feature tensors from the multispectral path and the panchromatic path are extracted respectively. The channel exchange module is used to perform cross-fusion in the channel dimension. The specific operation is as follows: By selecting some channels from the PAN features to replace the corresponding channels in the MS features, or by selecting some channels from the MS features to replace the corresponding channels in the PAN features, the initial injection and interaction of complementary structural information can be achieved. Step 4.2, Cross-modal fusion module: To further model cross-modal high-order representation information, a guided enhancement mechanism is adopted, which guides the fusion of panchromatic channel features by information-rich multispectral features. Specifically, this includes: The cross-fused features are fed into the cross-modal fusion module for further modeling. The MS channel features, which have richer information content, are used as guides. After linear mapping and SiLU activation, they are subjected to element-wise Hadamard product operation with the PAN channel features to realize the fusion guidance mechanism and enhance the collaborative expression between modalities.

6. The panchromatic sharpening method based on high-order state space modeling according to claim 5, characterized in that, Step 5 specifically includes: Step 5.1: Input the fused features into the image reconstruction module, which contains a 3×3 convolutional layer, to generate the output high-resolution multispectral image HrMS. Step 5.2: Introduce upsampled image MS through skip connections to improve detail restoration capabilities:

7. The panchromatic sharpening method based on high-order state space modeling according to claim 6, characterized in that, It also includes: Step 6, training the entire network structure through supervised learning, with the loss function including spatial error, structural similarity, and spectral consistency terms to achieve dual optimization of image sharpness and spectral fidelity; Step 6 specifically includes: Step 6.1, Spatial Domain Error Term: This term measures the overall error between the reconstructed image and the reference image in pixel space, using the mean absolute error as the metric. Among them, I HrMS To output an image to the network, I GT This is the corresponding high-resolution multispectral reference image; Step 6.2, Spectral Consistency Term: To maintain the spectral consistency of the fused image, a spectral angle mapping or scale-free relative global error metric is introduced as an additional regularization term to control spectral angle deviation and global error, using SAM: Among them, I HrMS To output an image to the network, I GT This is the corresponding high-resolution multispectral reference image; Step 6.3: The final combination of multinomial loss functions is a weighted sum, defined as follows: Among them, λ1 and λ2 are hyperparameters used to balance the impact weights of various losses.

8. A full-color sharpening system based on high-order state space modeling, characterized in that, The method used according to any one of claims 1 to 7 includes: The image preprocessing module is used to acquire pairs of remote sensing images with different resolutions, including low-resolution multispectral images and high-resolution panchromatic images, and to upsample the multispectral images to make their resolution consistent with that of the panchromatic images, thus obtaining the upsampled multispectral images. The feature extraction module is used to extract features from the upsampled multispectral image I. MS Compared with the original panchromatic image I PAN Shallow feature extraction is performed to obtain its initial feature representation, denoted as F. MS With F PAN ; A high-order state space modeling module construction module is used to construct a high-order state space modeling module. The extracted features are input into the high-order state space modeling module, which uses structured state space units to jointly model spatial and channel information. The modal-level dynamic fusion and high-order structure modeling module is used to enhance multispectral image I... MS With panchromatic image I PAN Cross-modal information interaction between them introduces a cross-modal fusion path, including a channel exchange module and a cross-modal fusion module, to achieve modal-level dynamic fusion and high-order structure modeling; The high-resolution multispectral image output module is used to input the fused features into the reconstruction module and output a high-resolution multispectral image.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are executed by a processor according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product stores computer instructions, which are executed by a processor according to any one of claims 1 to 7.

Images