Panchromatic sharpening method and system based on cross-resolution adversarial learning and Mama network

Abstract

The invention discloses a panchromatic sharpening method and system based on cross-resolution adversarial learning and a Mama network. The method comprises the following steps: acquiring a low-resolution multispectral image LRMS and a high-spatial-resolution panchromatic image PAN; extracting features of the low-resolution multispectral image LRMS to obtain a first feature map; extracting features of the high-spatial-resolution panchromatic image PAN to obtain a second feature map; performing wavelet decomposition and weighted fusion on the first feature map and the second feature map to obtain a multi-channel feature map, and performing dynamic fusion on the multi-channel feature map through a cross attention gating mechanism to obtain a fused feature map; and carrying out residual connection on the fused feature map and the low-resolution multispectral image, and then carrying out feature reconstruction to obtain a high-resolution multispectral remote sensing image. According to the method, the panchromatic sharpening performance of the remote sensing image can be improved, and the high-quality and high-resolution multispectral remote sensing image is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image full-color sharpening, and in particular relates to a full-color sharpening method and system based on cross-resolution adversarial learning and Mamba network. Background Art

[0002] Remote sensing imagery is data acquired through long-distance detection of the Earth's surface and atmosphere using sensors carried by platforms such as satellites and aircraft. Remote sensing images record electromagnetic radiation information from objects across different wavelengths, providing crucial data support for environmental monitoring, resource exploration, urban planning, and other fields.

[0003] High-resolution multispectral images (HRMS) are an important data type in remote sensing. They are generated by fusing low-resolution multispectral images (LRMS) with high-resolution panchromatic (PAN) images. Their goal is to simultaneously preserve the rich spectral information of multispectral images (used for object classification and environmental monitoring) and the high spatial resolution of panchromatic images (clearly displaying surface details). The technique for generating HRMS images is called pan-sharpening.

[0004] Among existing pan-sharpening technologies, traditional methods (such as those based on component replacement and multi-resolution analysis) often have difficulty achieving a balance between spectral fidelity and spatial detail, resulting in image distortion in the fusion results. Pan-sharpening methods based on deep learning are mainly divided into supervised and unsupervised categories. Supervised methods rely on reduced-resolution synthetic data for training, and their performance degrades significantly in full-resolution real scenes, showing poor generalization ability. Unsupervised methods, due to the lack of guidance from reference images, are easily affected by noise during training, resulting in difficulty in model convergence and possible spectral distortion in the generated results. Most existing deep learning methods have shortcomings in modeling the early interactions of multimodal (spectral and spatial) features. For example, simple splicing or shallow fusion cannot fully capture the deep cross-modal correlations. Although models based on Transformer or three-dimensional convolution can improve performance, they have a large number of parameters and slow inference speed. Summary of the Invention

[0005] To address the problems of supervised learning's dependence on reduced-resolution data, the difficulty of unsupervised methods in converging, and insufficient cross-modal feature interaction, this paper proposes a panchromatic sharpening method and system based on cross-resolution adversarial learning and Mamba network.

[0006] In a first aspect, the present invention proposes a panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network, the method comprising: collecting a remote sensing image set to obtain a low-resolution multispectral image LRMS and a high spatial resolution panchromatic image PAN;

[0007] The collected low-resolution multispectral image LRMS and high spatial resolution panchromatic image PAN are input into the trained remote sensing image pan-sharpening network to generate a high-resolution multispectral remote sensing image HRMS.

[0008] Furthermore, the network structure of the remote sensing image full-color sharpening network includes: a feature extraction module, a dynamic wavelet attention fusion module and a feature reconstruction module, and the feature reconstruction module adopts a generative adversarial network.

[0009] Furthermore, the trained remote sensing image pan-sharpening network is used to generate high-resolution multispectral remote sensing images (HRMS). The processing process includes:

[0010] S1: Extract features from the low-resolution multispectral image LRMS to obtain the first feature map; extract features from the high-spatial-resolution panchromatic image PAN to obtain the second feature map;

[0011] S2: performing wavelet decomposition and weighted fusion on the first feature map and the second feature map to obtain a multi-channel feature map, and dynamically fusing the multi-channel feature map through a cross-attention gating mechanism to obtain a fused feature map;

[0012] S3: Residual connection is performed on the fusion feature map and the low-resolution multispectral image LRMS, and then feature reconstruction is performed to obtain a high-resolution multispectral remote sensing image HRMS.

[0013] Furthermore, the structure of the dynamic wavelet attention fusion module includes a wavelet decomposition module, a low-frequency fusion module, a high-frequency enhancement module and a Mamba network, wherein the low-frequency fusion module adopts SENet.

[0014] Furthermore, the dynamic wavelet attention fusion module is used to perform weighted fusion on the extracted feature maps. The fusion process includes:

[0015] The first feature map extracted is subjected to wavelet decomposition to obtain a PAN low-frequency subband and a PAN high-frequency subband; the second feature map extracted is subjected to wavelet decomposition to obtain a LRMS low-frequency subband and a LRMS high-frequency subband;

[0016] SENet is used to perform weighted fusion of the LRMS low-frequency sub-band and the PAN low-frequency sub-band to obtain the low-frequency sub-band fusion feature map LL fused ;

[0017] Performing weighted fusion on the PAN high frequency sub-band and the LRMS high frequency sub-band to obtain a high frequency fusion result;

[0018] The low-frequency sub-band fusion feature map LL fused Combined with the high-frequency fusion result into a multi-channel feature map;

[0019] Convert the multi-channel feature map into a Mamba sequence F spectral The multi-channel feature map is processed by windowed Mamba to extract the spatial detail feature F spatial ;

[0020] Dynamically fusion of Mamba sequences via cross-attention gating mechanism spectral and spatial detail feature F spatial , and obtain the fused feature map after weighted fusion.

[0021] Furthermore, a fusion weight G is calculated according to the Mamba sequence and the spatial detail features. Based on the fusion weight G, the Mamba sequence and the spatial detail features are weightedly fused through a cross-attention gating mechanism to obtain a fusion feature map after weighted fusion.

[0022] Furthermore, the training phase of the remote sensing image pan-sharpening network is divided into three training phases, specifically:

[0023] In the first training stage, the remote sensing image pan-sharpening network is trained under supervision using reduced-resolution remote sensing image data.

[0024] In the second training stage, full-resolution remote sensing image data is used to perform unsupervised alignment training on the remote sensing image pan-sharpening network;

[0025] In the third training stage, the remote sensing image pan-sharpening network is trained using mixed down-resolution remote sensing image data and full-resolution remote sensing image data.

[0026] In a second aspect, the present invention provides a remote sensing image pan-sharpening system, the system comprising:

[0027] Remote sensing image acquisition module, used to acquire low-resolution multispectral images LRMS and high-spatial-resolution panchromatic images PAN;

[0028] A feature extraction module is used to extract feature maps of the collected remote sensing images;

[0029] A feature decomposition module, configured to decompose the feature map into low-frequency sub-bands and high-frequency sub-bands;

[0030] A dynamic wavelet attention fusion module is used to perform wavelet decomposition and weighted fusion on the low-frequency sub-band and the high-frequency sub-band to obtain a fusion feature;

[0031] The feature reconstruction module is used to reconstruct the fused features to obtain the HRMS image.

[0032] Beneficial effects of the present invention:

[0033] 1. The present invention designs a remote sensing image full-color sharpening network, in which the dynamic wavelet attention fusion module uses wavelet domain decomposition and cross-modal interaction to achieve adaptive fusion of spectral-spatial information, overcoming the high-frequency blurring and other problems caused by fixed weights in traditional wavelet methods.

[0034] 3. This paper designs a remote sensing image pan-sharpening network that balances spectral fidelity and spatial detail through frequency enhancement in the wavelet domain and long-range modeling using the Mamba network. To optimize cross-modal deep fusion, the Mamba network is used instead of the traditional Transformer. Leveraging Mamba's linear complexity, the training difficulty of the remote sensing image pan-sharpening network is reduced, breaking through the robustness bottleneck of traditional lightweight models in complex scenarios.

[0035] 3. This paper adopts a semi-supervised adversarial learning method to train the remote sensing image pan-sharpening network, combining the stability of supervised learning with the generalization ability of unsupervised learning. Through phased training, it reduces the dependence on full-resolution labels and alleviates the domain shift problem.

[0036] In summary, the present invention systematically solves the problems of spectral distortion, low computational efficiency and dynamic scene artifacts in the field of pan-sharpening, and generates high-quality high-resolution multispectral remote sensing images (HRMS). BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the network structure of the remote sensing image pan-sharpening network in an embodiment of the present invention;

[0038] Figure 2 This is a flowchart of the steps of generating a high-resolution multispectral remote sensing image HRMS using a remote sensing image pan-sharpening network in an embodiment of the present invention;

[0039] Figure 3 Schematic diagram of the network structure of SENet in an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of the network structure of the Mamba network in an embodiment of the present invention;

[0041] Figure 5 A comparison chart of simulation verification results of pan-sharpening remote sensing images using the present invention and existing algorithms;

[0042] Figure 6 Schematic diagram of the system structure of the remote sensing image full color sharpening system in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The terms "first," "second," "third," "fourth," and so on, in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate and are merely used to describe the manner in which objects with the same attributes are described in the embodiments of this application.

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] The embodiment of the present invention proposes a panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network, which includes: collecting a remote sensing image set to obtain a low-resolution multispectral image LRMS and a high-spatial-resolution panchromatic image PAN;

[0046] The collected low-resolution multispectral image LRMS and high spatial resolution panchromatic image PAN are input into the trained remote sensing image pan-sharpening network to output a high-resolution multispectral remote sensing image HRMS.

[0047] Figure 1 Schematic diagram of the network structure of the remote sensing image pan-sharpening network in an embodiment of the present invention. Figure 1 In the paper, the network structure of the remote sensing image pan-sharpening network includes: a feature extraction module, a dynamic wavelet attention fusion module, and a feature reconstruction module. Among them, the feature reconstruction module adopts a generative adversarial network. The input of this network model is a high-spatial-resolution panchromatic image PAN and a low-resolution multispectral image LRMS. In the model, Conv represents convolution, LeakyReLU represents the LeakyReLU activation function, SENet (Squeeze-and-Excitation Networks) is a convolutional neural network model for image classification, Mamba represents a linear time series network (model) based on the selection state space, and U-net represents a U-type network. The output of this model is a high-resolution multispectral remote sensing image HRMS.

[0048] In the illustrated embodiment, reference is made to Figure 1 As shown, the network structure of the remote sensing image full-color sharpening network includes: a feature extraction module, a dynamic wavelet attention fusion module and a feature reconstruction module, and the feature reconstruction module adopts a generative adversarial network.

[0049] Figure 2 This is a flowchart of the steps of generating a high-resolution multispectral remote sensing image HRMS using a remote sensing image pan-sharpening network in an embodiment of the present invention.

[0050] In the illustrated embodiment, reference is made to Figure 2 As shown in Figure 2, the trained remote sensing image pan-sharpening network is used to generate high-resolution multispectral remote sensing images HRMS. The processing process includes:

[0051] S1: Acquire low-resolution multispectral images LRMS and high-spatial-resolution panchromatic images PAN respectively.

[0052] Specifically, sensors aboard platforms such as satellites and aircraft conduct long-range surveys of the Earth's surface and atmosphere, collecting remote sensing imagery. For example, remote sensing imagery is collected by the United States' fourth-generation, high-resolution commercial remote sensing satellite, WorldView-III, and by China's civilian optical remote sensing satellite, GaoFen-2. When different sensors on remote sensing satellites capture remote sensing images of the same target at the same location at the same time, they can produce a low-resolution multispectral (LRMS) image and a high-spatial-resolution panchromatic (PAN) image of the same target.

[0053] S2: Extract features from the low-resolution multispectral image LRMS to obtain the first feature map; extract features from the high-spatial-resolution panchromatic image PAN to obtain the second feature map.

[0054] Specifically, a residual dense block is used to extract features from the high-spatial-resolution panchromatic image (PAN) to obtain the first feature map. The network structure of the residual dense block includes a convolutional layer (Conv) and an activation layer (Leaky ReLU activation function). The low-resolution multispectral image (LRMS) is upsampled to obtain the second feature map.

[0055] S3: Perform wavelet decomposition and weighted fusion on the first feature map and the second feature map to obtain a multi-channel feature map, and dynamically fuse the multi-channel feature map through a cross-attention gating mechanism to obtain a fused feature map.

[0056] In the illustrated embodiment, reference is made to Figure 2 As shown, an embodiment of the present invention designs a remote sensing image pan-sharpening network based on cross-resolution adversarial learning and a Mamba network. The network includes a feature extraction module, a dynamic wavelet attention fusion module, and a feature reconstruction module. The structure of the dynamic wavelet attention fusion module includes a wavelet decomposition module, a low-frequency fusion module, a high-frequency enhancement module, and a Mamba network, where the low-frequency fusion module adopts SENet.

[0057] Specifically, the wavelet decomposition module is used to perform wavelet decomposition on the input feature map to extract high-resolution features and low-resolution features. The low-frequency fusion module (specifically SENet) is used to perform low-frequency fusion on the low-resolution features to obtain a low-frequency fusion feature map. The high-resolution features are weightedly fused to obtain a high-frequency fusion result. The Mamba network is used to perform cross-modal interactive fusion on the high-frequency fusion result and the low-frequency fusion features to obtain a multi-channel feature map.

[0058] It should be noted that wavelet decomposition refers to the Discrete Wavelet Transform (DWT). MATLAB software is typically used to implement wavelet decomposition of the input (feature map), extracting both coarse-scale (low-resolution) and fine-scale (high-resolution) features from the input (feature map). Coarse-scale (low-resolution) features refer to the low-frequency portion of the input (feature map), capturing its primary, broad characteristics. Fine-scale (high-resolution) features refer to the high-frequency portion of the input (feature map), capturing its details and rapid changes.

[0059] It should be noted that SENet (Squeeze-and-Excitation Networks) is a convolutional neural network model for image classification. Its core idea is to explicitly model the dependencies between channels. By adaptively adjusting the feature map weights of each channel, the model can pay more attention to feature channels with large amounts of information while suppressing unimportant feature channels.

[0060] It should be noted that the Mamba (Linear-Time Sequence Modeling with Selective State Space) network is a new linear time series network model based on the Selective State Space Model (SSM). It combines the characteristics of recurrent neural networks (RNNs) and convolutional neural networks (CNNs). Through recursion or convolution operations, it achieves linear or near-linear scaling of computational cost with sequence length, significantly reducing computational complexity.

[0061] Figure 3 Schematic diagram of the network structure of SENet in an embodiment of the present invention. Figure 3In the figure, Inception represents interpolation. In the example of the present invention, bicubic interpolation is used for upsampling, GlobalPooling represents global pooling, FC represents a fully connected layer, ReLU represents a ReLU activation function, Sigmoid represents a Sigmoid activation function, and Scale represents a channel weight scaling operation. Specifically, the channel attention weight (with a value range of [0, 1]) output by the Sigmoid activation function is multiplied by the original input feature map channel by channel.

[0062] In the illustrated embodiment, reference is made to Figure 3 As shown in the figure, the network structure of SENet includes: input layer, Inception layer, pooling layer, first fully connected layer (ReLU activation function), second fully connected layer (Sigmoid activation function), Scale layer and output layer.

[0063] The Inception layer represents interpolation of an image. Specifically, in the example shown, bicubic interpolation is used for upsampling.

[0064] FC layers are fully connected layers. In the example shown, the first fully connected layer (FC) reduces the number of channels from C to C / r (r is the compression ratio, in this example, r = 4), reducing the computational effort through dimensionality reduction. The second fully connected layer then restores the number of channels from C / r to C, preparing for the subsequent generation of channel attention weights.

[0065] The Scale layer represents the channel weight scaling operation. Its specific function is to multiply the channel attention weight (value range is [0,1]) output by the Sigmoid activation function with the original input feature map channel by channel.

[0066] Figure 4 Schematic diagram of the network structure of the Mamba network in an embodiment of the present invention. Figure 4 In

[15] , the network structure of the Mamba network includes a layer normalization layer (Norm), the output of which is connected to two branches. One branch includes the first multilayer perceptron (MLP), a convolutional layer (Conv1d) and a state-space model (SSM), and the other branch includes the second multilayer perceptron (MLP) and an activation layer (ReLU activation function). The output results of the two branches are multiplied and input into the third multilayer perceptron (MLP). The output result of the third multilayer perceptron (MLP) is residually connected with the original input to obtain the final output of the Mamba network.

[0067] In the illustrated embodiment, the extracted feature maps are weightedly fused using a dynamic wavelet attention fusion module, and the fusion process includes:

[0068] S301: performing wavelet decomposition on the extracted first feature map to obtain a PAN low-frequency sub-band and a PAN high-frequency sub-band; performing wavelet decomposition on the extracted second feature map to obtain a LRMS low-frequency sub-band and a LRMS high-frequency sub-band.

[0069] Specifically, the first feature map is decomposed using the Haar wavelet basis to obtain the LRMS low-frequency subband (LL LRMS ), LRMS high frequency sub-band (LH LRMS / HL LRMS / HH LRMS ). LH LRMS Represents the low-frequency subband of the LRMS image along the horizontal direction, HL PAN Represents the low-frequency subband of the LRMS image along the vertical direction, HH PAN Represents the low-frequency subband of the LRMS image along the diagonal direction.

[0070] The second feature map is decomposed using Haar wavelet basis to obtain the PAN low frequency subband (LL PAN ) and PAN high frequency sub-band (LH PAN / HL PAN / HH PAN ). LH PAN Represents the high-frequency sub-band of the PAN image along the horizontal direction, HL PAN Indicates the high-frequency subband of the PAN image along the vertical direction, HH PAN Represents the high-frequency sub-band of the PAN image along the diagonal direction.

[0071] S302: Use SENet to perform weighted fusion on the LRMS low-frequency sub-band and the PAN low-frequency sub-band to obtain a low-frequency sub-band fusion feature map LL fused .

[0072] Specifically, refer to Figure 2 、 Figure 3 As shown in the figure, the LRMS low-frequency subband and the PAN low-frequency subband are input into SENet respectively to obtain the corresponding weight values, and the LRMS low-frequency subband and the PAN low-frequency subband are weighted fused according to the obtained weights to output the low-frequency subband fusion feature map LL fused .

[0073] The low-frequency sub-band fusion feature map LL fused Expressed as:

[0074] LL fused =SENet(LL LRMS )·LL LRMS +(1-SENet(LL PAN ))·LL PAN

[0075] Where, LL fused Represents the low-frequency sub-band fusion feature map, SENet(.) represents the SENet network, LL LRMS Represents the LRMS low-frequency subband, LL PAN Indicates the PAN low-frequency sub-band.

[0076] The low-frequency subband fusion feature map LLfused generated by SENet channel attention weighted fusion can adaptively allocate LRMS and PAN low-frequency subband weights according to channel importance, realize adaptive fusion of spectral-spatial information, effectively balance spectral fidelity and spatial details, and avoid spectral distortion or lack of details caused by traditional fixed weight fusion.

[0077] S303: Perform weighted fusion on the PAN high frequency sub-band and the LRMS high frequency sub-band to obtain a high frequency fusion result.

[0078] Specifically, the gradient amplitude of the PAN image is calculated and used as the complexity index. Based on the complexity index, the convolutional network is used to perform weighted fusion of the PAN high-frequency sub-band and the LRMS high-frequency sub-band to obtain the high-frequency fusion result.

[0079] The gradient amplitude of the PAN image is calculated as the complexity index, and its calculation formula is:

[0080]

[0081] In the formula, G represents the complexity index, Represents the high frequency subband of the PAN image (LH PAN / HL PAN / HH PAN ) in the horizontal The gradient of the direction, Represents the high frequency subband of the PAN image (LH PAN / HL PAN / HH PAN ) in vertical Directional gradient.

[0082] The complexity index G is used as the input of the subsequent lightweight convolutional network to obtain the weights of each high-frequency sub-band.

[0083] Through a lightweight convolutional network (for example, a simple convolutional network with two layers of 1×1 convolution and ReLU activation function), the weights of each high-frequency sub-band are learned and weighted fusion of the high-frequency sub-bands is performed:

[0084] w LH =ConvBlock(G),

[0085] LH fused=w LH ·LH PAN ;

[0086] Where w LH Represents the LH obtained through lightweight network learning PAN Sub-band weight, ConvBlock(.) represents a lightweight convolutional network, LH fused Represents the high-frequency subband LH of the PAN image along the horizontal direction PAN The weighted result of LH PAN Represents the high-frequency sub-band of the PAN image along the horizontal direction.

[0087] w HL =ConvBlock(G),

[0088] HL fused =w HL ·HL PAN ;

[0089] Where w HL Represents the HL obtained by lightweight network learning PAN Sub-band weights, ConvBlock(.) represents a lightweight convolutional network, HL fused Represents the high-frequency sub-band HL of the PAN image along the horizontal direction PAN The weighted result of HL PAN Represents the high-frequency sub-band of the PAN image along the vertical direction.

[0090] w HH =ConvBlock(G),

[0091] HH fused =w HH ·HH PAN

[0092] Where w HH Represents HH learned through lightweight network PAN The weight of the sub-band, ConvBlock (.) represents a lightweight convolutional network, HH fused Represents the high-frequency subband HH of the PAN image along the diagonal direction PAN The weighted result, HH PAN Represents the high-frequency sub-band of the PAN image along the diagonal direction.

[0093] In step S303, high-frequency sub-bands are fused based on the dynamic learning weights of the PAN gradient amplitude. The fusion strength can be adaptively adjusted according to the image complexity to optimize the clarity of spatial details (edges, textures), avoid blurring or noise amplification, and enhance cross-modal high-frequency feature interaction, reduce spectral distortion, and improve structural consistency.

[0094] S304: low frequency fusion result LL fused Compared with the high frequency fusion results (LH fused , HL fused , HH fused ) are spliced ​​into a multi-channel feature map.

[0095] In step S304, the low-frequency fusion results and the high-frequency fusion results are spliced ​​into a multi-channel feature map, which can integrate the low-frequency global spectral-spatial fusion information and the high-frequency local detail information to form a multi-scale feature representation, providing rich input for subsequent Mamba modeling, facilitating cross-modal deep interaction and dynamic fusion, enhancing feature expression capabilities, helping to more comprehensively capture the image spectral correlation and spatial structure, and improving the balance between the spectral fidelity and spatial resolution of the fused image.

[0096] S305: Convert the multi-channel feature map into a Mamba sequence F spectral The multi-channel feature map is processed by windowed Mamba to extract the spatial detail feature F spatial .

[0097] Specifically, the multispectral channel map is regarded as a sequence, and Mamba sequence modeling is performed along the multispectral channel dimension to capture the long-range dependency between bands, and the Mamba sequence F is obtained. spectral .

[0098] The Mamba Sequence F spectral Specifically expressed as:

[0099] F spectral =Mamba(LL fused ,axis=C)

[0100] Where, F spectral represents the Mamba sequence, that is, the result of Mamba sequence modeling of the low-frequency sub-band fusion feature map along the multi-spectral channel dimension, Mamba(.) represents the Mamba network, LL fused Represents the low-frequency sub-band fusion feature map, axis = C indicates processing along the channel dimension, and C represents the channel dimension.

[0101] In this step, by treating the multispectral channels as sequences and performing Mamba sequence modeling along the channel dimension to capture the long-range spectral dependence, the deep correlation between multispectral bands can be explored, avoiding the problem of insufficient modeling of spectral correlation in traditional methods, thereby improving the spectral consistency of the fused image and enhancing the model's global modeling ability for complex spectral features, ultimately achieving more accurate spectral fidelity.

[0102] An 8×8 windowed Mamba process is performed on the high frequency subband in the multi-channel feature map to extract spatial detail features.

[0103] The spatial detail feature is specifically expressed as:

[0104] F spatial =WindowMamba(Concat(LH fused ,HL fused ,HH fused ))

[0105] Where, F spatial Represents the spatial detail features of the high-frequency sub-band processed by windowed Mamba, WindowMamba(.) represents the windowed Mamba operation, Concat(.) represents the concatenation of feature maps along the channel dimension, LH PAN Represents the high-frequency sub-band of the PAN image along the horizontal direction, HL PAN Indicates the high-frequency subband of the PAN image along the vertical direction, HH PAN Represents the high-frequency sub-band of the PAN image along the diagonal direction.

[0106] The windowed Mamba operation processes the concatenated high-frequency sub-band feature set with a set window size (for example, the window size is set to 8×8). By performing Mamba operations within each window, the Mamba model can focus on local areas and effectively extract spatial detail features in these high-frequency sub-bands.

[0107] S306: Dynamically fusion of Mamba sequences via cross-attention gating mechanism spectral and spatial detail feature F spatial , and obtain the fused feature map after weighted fusion.

[0108] Specifically, according to Mamba sequence F spectral and spatial detail feature F spatial , calculate the fusion weight G, and its calculation formula is:

[0109] G=Sigmoid(W g ·[F spectral ,F spatial ])

[0110] Among them, G represents the calculated fusion weight, Sigmoid(.) represents the Sigmoid activation function, and W g is the learnable parameter matrix, F spectral Represents the Mamba sequence F spectral , that is, the result of Mamba sequence modeling of low-frequency fusion features along the multi-spectral channel dimension, F spatial Represents the spatial detail feature, that is, the result of windowed Mamba processing on the high-frequency sub-band.

[0111] Based on the fusion weight G, the Mamba sequence F is spectral and spatial detail feature F spatial Perform weighted fusion to obtain a fusion feature map after weighted fusion.

[0112] F fusion =G⊙F spectral +(1-G)⊙F spatial

[0113] Among them, F fusion Represents the fusion feature map after weighted fusion, that is, the Mamba sequence F spectral and spatial detail feature F spatial The result is dynamically fused through the cross-attention gating mechanism, and ⊙ represents element-wise multiplication.

[0114] S4: Residual connection is performed on the fusion feature map and the low-resolution multispectral image LRMS, and then feature reconstruction is performed to obtain a high-resolution multispectral remote sensing image HRMS.

[0115] Specifically, for the fusion feature map F fusion Perform wavelet reconstruction and restore it to a spatial domain image, which is specifically expressed as:

[0116] HRMS base =IDWT(F fusion )

[0117] Among them, HRMS base Represents the fusion result F fusion The spatial domain image is restored through wavelet reconstruction, IDWT(·) represents the inverse wavelet transform, F fusion Represents the fusion feature map after weighted fusion, that is, the Mamba sequence F spectral and spatial detail feature F spatial The results are dynamically fused through the cross-attention gating mechanism.

[0118] HRMS spatial domain image base After residual connection with the original low-resolution multispectral image LRMS, it is input into the trained generator for feature reconstruction to obtain the high-resolution multispectral remote sensing image HRMS, which is specifically expressed as:

[0119] HRMS final =HRMS base +UpSample(LRMS)

[0120] Among them, HRMS final represents the high-resolution multispectral remote sensing image HRMS, which is the fusion result finally output by the model. UpSample(·) represents upsampling.

[0121] The fused feature map and the LRMS image are connected using residual connections to avoid information loss and enhance the spectral fidelity of the generated image.

[0122] In the illustrated embodiment, during the training phase of the entire network (i.e., the remote sensing image pan-sharpening network), the embodiment of the present invention adopts a three-stage training process, wherein the training process includes:

[0123] The first training phase: The entire network is trained under supervision using the reduced-resolution remote sensing image data. In the first training phase, the optimizer uses Adam (lr = 3e-4, β1 = 0.5, β2 =

[0124] 0.999). lr represents the learning rate, e represents the symbol of scientific notation, and here 3e-4 is equivalent to 3×10 -4 , β1 represents the exponential decay rate of the first-order moment estimate, and β2 represents the exponential decay rate of the second-order moment estimate.

[0125] The loss function of the first training stage is designed as follows:

[0126] (1) Joint confrontation losses Using the idea of ​​generative adversarial networks, the generator attempts to generate realistic fusion images, while the discriminator determines whether the generated images are real or generated. The specific calculation formula is:

[0127]

[0128] Among them, x represents the real down-resolution fusion image, z represents the noise input to the generator, D(x) represents the output of the discriminator for the real image, D(G(z)) represents the output of the discriminator for the generated image, and p real represents the real data distribution, p fake Represents the generated data distribution.

[0129] Through adversarial training, the generator is encouraged to generate fused images that are closer to reality.

[0130] Spectral angle loss It is used to measure the similarity of the spectral characteristics between the generated fusion image and the real high-resolution multispectral image (HRMS). The calculation formula is:

[0131]

[0132] Where N is the total number of image pixels, F i is the spectrum vector of the generated image at the i-th pixel, Y iis the spectral vector of the true HRMS image at pixel i. The smaller the spectral angle loss, the more similar the spectral characteristics of the generated image are to the true image, which helps to ensure the spectral fidelity of the fused image.

[0133] Wavelet high frequency L1 loss Wavelet transform is used to decompose the image into sub-bands of different frequencies, and L1 loss is calculated for the high-frequency sub-bands.

[0134] Specifically, the generated fusion image and the real HRMS image are first subjected to wavelet transform to obtain high-frequency sub-bands, and then the L1 distance between them is calculated. The wavelet high-frequency L1 loss The specific calculation formula is:

[0135]

[0136] Among them, M is the total number of all high-frequency sub-band pixels, F j (m,n) is the pixel value of the j-th high-frequency subband at the (m,n) position of the generated image, Y j (m,n) is the pixel value of the jth high-frequency subband at position (m,n) of the real HRMS image. This loss term can encourage the model to better preserve the high-frequency details of the image.

[0137] The overall loss function of the first training stage is specifically:

[0138]

[0139] in, represents the overall loss of the first training stage, represents the overall loss of the first training stage, Indicates joint confrontation loss, is the spectral angle loss, is the wavelet high frequency L1 loss, λ adv represents the first hyperparameter, λ sam represents the second hyperparameter, λ high Represents the third hyperparameter, which is used to balance the importance of different loss terms.

[0140] In the early stage of the first training phase, λ can be appropriately increased sam The weight of λ makes the entire network model pay more attention to the learning of spectral fidelity; as the training progresses, gradually adjust λ adv and λ high The weights of , allow the entire network model to better learn to generate adversarial and high-frequency details while ensuring spectral fidelity.

[0141] Second training phase: The entire network is trained with unsupervised alignment using full-resolution remote sensing image data. The loss function for the second training phase is designed as follows:

[0142] (1) Translation-invariant spectral loss: In order to avoid the problems of traditional spectral loss when processing misaligned areas, a translation-invariant calculation method is adopted. This is achieved by constructing a spectral similarity measure based on a sliding window. A fixed-size window (a 3×3 window is used in the example of the present invention) is slid on the generated image and the true full-resolution image, and the similarity of the spectral vector in each window is calculated. Assuming the window size is (s×s) and the position on the image is (i, j), the translation-invariant spectral loss calculation formula is:

[0143]

[0144] in, represents the translation-invariant spectral loss, P is the total number of sliding windows, (i, j) represents the starting position coordinates of the sliding window on the image, (k, l) represents the translation amount within the window, window refers to the sliding window, which is a fixed-size area, and F i+m,j+n Indicates the spectral vector of the pixel at (m,n) in the window starting at (i,j) in the generated image. i+k+m,j+l+m Represents the spectral vector of the pixel at coordinate (m,n) in the window after being shifted by (k,l) relative to the position (i,j) in the true full-resolution image.

[0145] By calculating the minimum spectral angle under all possible translations, the translation-invariant spectral loss is obtained, which effectively suppresses artifacts in misaligned areas.

[0146] (2) Cross-modal contrast loss: Cross-modal contrast loss aims to enhance the alignment between spectral features and spatial features. The spectral features and spatial features of the generated image are regarded as two modalities respectively, and positive sample pairs and negative sample pairs are constructed. The spectral and spatial features at the same position are regarded as positive sample pairs, while the spectral and spatial features at different positions are regarded as negative sample pairs. The calculation formula of the cross-modal contrast loss is:

[0147]

[0148] Among them, Q represents the total number of sample pairs, s q represents the spectral feature vector in the qth positive sample pair, t q represents the spatial feature vector in the qth positive sample pair, and τ is a hyperparameter used to adjust the strength of the contrast loss. By maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs, the model is encouraged to better integrate spectral and spatial features.

[0149] The overall loss function calculation formula for the second training stage is:

[0150]

[0151] in, represents the overall loss of the second training stage, represents the translation-invariant spectral loss, represents the cross-modal contrast loss, λ spectral represents the first weight coefficient, λ contrast represents the second weight coefficient, λ spectral ∈[0,1],λ contrast ∈[0,1], and λ spectral +λ contrast =1.

[0152] By introducing translation-invariant spectral loss and cross-modal contrast loss to construct the overall loss function of the second training stage, it can be used to suppress artifacts in misaligned areas during the second stage training of the entire network model, so that the entire network model can better adapt to full-resolution remote sensing image data in real scenes.

[0153] The third training phase: The entire network is trained using a mixture of down-scaled and full-resolution remote sensing image data. During training, each batch contains eight pairs of down-scaled and full-resolution remote sensing image data, dynamically balancing supervised and unsupervised losses.

[0154] During the training process, the weights of supervised loss and unsupervised loss are dynamically adjusted according to the training round. Design a weight adjustment function that changes with the training round t:

[0155] For the supervision loss weight α(t), its calculation formula is:

[0156]

[0157] Among them, α(t) represents the supervision loss weight, T1 represents a preset training round turning point, for example, T1 = 30, and γ represents the adjustment rate parameter, for example, γ = 5.

[0158] In the early stages of training, α(t) is small and the model focuses more on learning unsupervised loss; as the number of training rounds increases, α(t) gradually increases and the weight of supervised loss increases.

[0159] For unsupervised loss weights, the weight is set to 1-α(t).

[0160] In the third training stage, by dynamically adjusting the supervised loss and unsupervised loss, the entire network model can fully utilize the advantages of different types of data during training and balance the stability and generalization of training.

[0161] Based on the same inventive concept, an embodiment of the present invention also proposes a full-color sharpening system based on cross-resolution adversarial learning and Mamba network. This system has the same or similar technical features as the above-mentioned full-color sharpening method based on cross-resolution adversarial learning and Mamba network. The same or similar technical features will not be repeated below.

[0162] Reference Figure 5 As shown, the full color sharpening system based on cross-resolution adversarial learning and Mamba network includes:

[0163] Remote sensing image acquisition module, used to acquire low-resolution multispectral images LRMS and high-spatial-resolution panchromatic images PAN;

[0164] A feature extraction module is used to extract feature maps of the collected remote sensing images;

[0165] A feature decomposition module, configured to decompose the feature map into low-frequency sub-bands and high-frequency sub-bands;

[0166] A dynamic wavelet attention fusion module is used to perform wavelet decomposition and weighted fusion on the low-frequency sub-band and the high-frequency sub-band to obtain a fusion feature;

[0167] The feature reconstruction module is used to reconstruct the fused features to obtain the HRMS image.

[0168] Experimental verification:

[0169] Comparative experiments and ablation experiments are designed to demonstrate the effectiveness of each module in the remote sensing image pan-sharpening network proposed in the embodiment of the present invention and the effectiveness of the entire method. The evaluation indicators include spectral angle mapping (SAM), global relative error (ERGAS), root mean square error (RMSE), spatial correlation coefficient (SCC), and general image quality extended to n-band quality (Q). n Ablation experiments were designed and conducted under four conditions: no wavelet-Mamba interaction, no dynamic weight scheduling, no cross-modal contrast loss, and a complete model to verify the effectiveness of each module mentioned in this invention. Table 1 shows the comparative experimental results of this experiment. Table 2 shows the ablation experimental results of this experiment.

[0170] Table 1 Comparative experimental results

[0171]

[0172] In Table 1, PNN (Pansharpening by convolutional neural networks) indicates that the PNN algorithm is used to perform pan-sharpening on remote sensing images, PanNet (A deep network architecture for pan-sharpening) indicates that the PanNet algorithm is used to perform pan-sharpening on remote sensing images, PanCSC (A Model-Driven DeepUnfolding Method for Pan-sharpening) indicates that the PanCSC algorithm is used to perform pan-sharpening on remote sensing images, and GPPNN (Deep Gradient Projection Networks for Pan-sharpening) indicates that the GPPNN algorithm is used to perform pan-sharpening on remote sensing images.

[0173] Figure 5 The following is a comparison chart of simulation verification results of using the present invention and the existing algorithm to perform full color sharpening on remote sensing images. Figure 5 In the figure, it can be seen that the overall spectral quality of the fused region is well maintained for the PNN, PanNet, PanCSC, and GPPNN methods, but the spectral details at the edges lack continuity, resulting in spectral distortion. The PanCSC method also exhibits significant artifacts. In contrast, the proposed method not only restores better spectral detail but also reconstructs more complete spatial details.

[0174] Table 2 Ablation experiment results

[0175]

[0176] In Table 2, A represents no wavelet-Mamba interaction, B represents no dynamic weight scheduling, C represents no cross-modal contrast loss, and D represents the complete remote sensing image pan-sharpening network.

[0177] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: ROM, RAM, disk or CD, etc.

[0178] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A full color sharpening method based on cross-resolution adversarial learning and Mamba network, characterized in that: include: Collect remote sensing image sets to obtain low-resolution multispectral images LRMS and high-spatial-resolution panchromatic images PAN; The collected low-resolution multispectral image LRMS and high spatial resolution panchromatic image PAN are input into the trained remote sensing image pan-sharpening network to generate a high-resolution multispectral remote sensing image HRMS.

2. The method for full color sharpening based on cross-resolution adversarial learning and Mamba network according to claim 1, characterized in that: The network structure of the remote sensing image full color sharpening network includes: a feature extraction module, a dynamic wavelet attention fusion module and a feature reconstruction module, and the feature reconstruction module adopts a generative adversarial network.

3. The method for full color sharpening based on cross-resolution adversarial learning and Mamba network according to claim 1, characterized in that: The trained remote sensing image pan-sharpening network is used to generate high-resolution multispectral remote sensing images (HRMS). The processing process includes: Feature maps are extracted from low-resolution multispectral images LRMS and high-spatial-resolution panchromatic images PAN respectively; The extracted feature maps are weighted fused to obtain a fused feature map with enhanced spatial and spectral information; The fused feature map after spatial and spectral information enhancement is residually connected with the LRMS image and then feature reconstruction is performed to obtain a high-resolution multispectral remote sensing image HRMS.

4. The method for full color sharpening based on cross-resolution adversarial learning and Mamba network according to claim 2, characterized in that: The structure of the dynamic wavelet attention fusion module includes a wavelet decomposition module, a low-frequency fusion module, a high-frequency enhancement module and a Mamba network, wherein the low-frequency fusion module adopts SENet.

5. The method for full color sharpening based on cross-resolution adversarial learning and Mamba network according to claim 3, characterized in that: The dynamic wavelet attention fusion module is used to perform weighted fusion on the extracted feature maps. The fusion process includes: Performing wavelet decomposition on the first feature map to obtain a PAN low-frequency subband and a PAN high-frequency subband; performing wavelet decomposition on the second feature map to obtain an LRMS low-frequency subband and an LRMS high-frequency subband; Performing weighted fusion on the LRMS low-frequency sub-band and the PAN low-frequency sub-band to obtain a low-frequency sub-band fusion feature map; Performing weighted fusion on the PAN high frequency sub-band and the LRMS high frequency sub-band to obtain a high frequency fusion result; The low-frequency sub-band fusion feature map and the high-frequency fusion result are spliced ​​into a multi-channel feature map; Converting the multi-channel feature map into a Mamba sequence, and subjecting the multi-channel feature map to windowed Mamba processing to extract spatial detail features; The Mamba sequence and spatial detail features are dynamically fused through the cross-attention gating mechanism to obtain a weighted fused feature map.

6. The method for full color sharpening based on cross-resolution adversarial learning and Mamba network according to claim 5, characterized in that: The Mamba sequence is specifically expressed as: F spectral =Mamba(LL fused ,axis=C) Where, F spectral Indicates Mamba sequence, Mamba(.) indicates Mamba network, LL fused Represents the low-frequency sub-band fusion feature map, axis = C indicates processing along the channel dimension, and C represents the channel dimension.

7. The method for full color sharpening based on cross-resolution adversarial learning and Mamba network according to claim 5, characterized in that: The spatial detail feature is specifically expressed as: F spatial =WindowMamba(Concat(LH fused ,HL fused ,HH fused )) Where, F spatial Represents the spatial detail features of the high-frequency subband processed by windowed Mamba, WindowMamba(.) represents the windowed Mamba operation, Concat(.) represents the concatenation of feature maps along the channel dimension, LH PAN Represents the high-frequency sub-band of the PAN image along the horizontal direction, HL PAN Represents the high-frequency subband of the PAN image along the vertical direction, HH PAN Represents the high-frequency sub-band of the PAN image along the diagonal direction.

8. The method for full color sharpening based on cross-resolution adversarial learning and Mamba network according to claim 5, characterized in that: According to the Mamba sequence and the spatial detail features, a fusion weight G is calculated. Based on the fusion weight G, the Mamba sequence and the spatial detail features are weightedly fused through a cross-attention gating mechanism to obtain a fusion feature map after weighted fusion.

9. The method for full color sharpening based on cross-resolution adversarial learning and Mamba network according to claim 1, characterized in that: The training phase of the remote sensing image pan-sharpening network is divided into three training phases, specifically: In the first training stage, the remote sensing image pan-sharpening network is trained under supervision using reduced-resolution remote sensing image data. In the second training stage, full-resolution remote sensing image data is used to perform unsupervised alignment training on the remote sensing image pan-sharpening network; In the third training stage, the remote sensing image pan-sharpening network is trained using mixed down-resolution remote sensing image data and full-resolution remote sensing image data.

10. A remote sensing image full color sharpening system, characterized in that: include: Remote sensing image acquisition module, used to acquire low-resolution multispectral images LRMS and high-spatial-resolution panchromatic images PAN; A feature extraction module is used to extract feature maps of the collected remote sensing images; A feature decomposition module, configured to decompose the feature map into low-frequency sub-bands and high-frequency sub-bands; A dynamic wavelet attention fusion module is used to perform wavelet decomposition and weighted fusion on the low-frequency sub-band and the high-frequency sub-band to obtain a fusion feature; The feature reconstruction module is used to reconstruct the fused features to obtain a high-resolution multispectral image.

Images