Panchromatic sharpening method and system based on cross-resolution adversarial learning and mamba network

Abstract

The application discloses a panchromatic sharpening method and system based on cross-resolution adversarial learning and a Mamba network. The method comprises the following steps: acquiring a low-resolution multispectral image LRMS and a high-spatial-resolution panchromatic image PAN; extracting features from the low-resolution multispectral image LRMS to obtain a first feature map; extracting features from the high-spatial-resolution panchromatic image PAN to obtain a second feature map; performing wavelet decomposition and weighted fusion on the first feature map and the second feature map to obtain a multi-channel feature map; performing dynamic fusion on the multi-channel feature map through a cross-attention gate mechanism to obtain a fused feature map; and performing feature reconstruction on the fused feature map and the low-resolution multispectral image after residual connection to obtain a high-resolution multispectral remote sensing image. The application can improve the performance of panchromatic sharpening of a remote sensing image and obtain a high-quality high-resolution multispectral remote sensing image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image panchromatic sharpening, and particularly relates to a panchromatic sharpening method and system based on cross-resolution adversarial learning and a Mamba network. BACKGROUND

[0002] Remote sensing images are image data obtained by remotely detecting the earth's surface and atmosphere through sensors carried by platforms such as satellites and aircraft. Remote sensing images can record electromagnetic wave radiation information of ground objects at different wavebands, providing important data support for fields such as environmental monitoring, resource exploration, and urban planning.

[0003] High-resolution multi-spectral images (HRMS) are an important data type in the field of remote sensing, which are generated by fusing low-resolution multi-spectral images (LRMS) and high spatial resolution panchromatic (PAN) images, aiming to retain both the rich spectral information of multi-spectral images (for ground object classification, environmental monitoring, etc.) and the high spatial resolution of panchromatic images (to clearly display surface details). The technology for generating HRMS images is called panchromatic sharpening.

[0004] In existing panchromatic sharpening techniques, traditional methods (such as those based on component substitution and multi-resolution analysis) often struggle to balance spectral fidelity and spatial detail, leading to image distortion in the fusion results. Deep learning-based panchromatic sharpening methods are mainly divided into supervised and unsupervised categories. Supervised methods rely on down-resolution synthetic data for training, and their performance significantly decreases in full-resolution real scenes, showing poor generalization ability. Unsupervised methods, lacking the guidance of reference images, are easily disturbed by noise during training, making it difficult for the model to converge, and the generated results may appear spectrally distorted. Most existing deep learning methods have deficiencies in early-stage interaction modeling of multi-modal (spectral and spatial) features, for example, simple concatenation or shallow fusion cannot fully capture deep cross-modal associations. Although models based on Transformers or three-dimensional convolution can improve performance, they have a large number of parameters and slow inference speed. SUMMARY

[0005] To address the dependence of supervised learning on down-resolution data, the convergence difficulties of unsupervised methods, and the insufficient interaction of cross-modal features, the present application proposes a panchromatic sharpening method and system based on cross-resolution adversarial learning and a Mamba network.

[0006] In a first aspect, the present application provides a panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network, which comprises: collecting a remote sensing image set, and respectively acquiring a low-resolution multispectral image LRMS and a high spatial resolution panchromatic image PAN;

[0007] inputting the collected low-resolution multispectral image LRMS and high spatial resolution panchromatic image PAN into the trained remote sensing image panchromatic sharpening network to generate a high-resolution multispectral remote sensing image HRMS.

[0008] Further, the network structure of the remote sensing image panchromatic sharpening network comprises: a feature extraction module, a dynamic wavelet attention fusion module and a feature reconstruction module, and the feature reconstruction module adopts a generative adversarial network.

[0009] Further, the high-resolution multispectral remote sensing image HRMS is generated by using the trained remote sensing image panchromatic sharpening network, and the processing process comprises:

[0010] S1: extracting a feature map from the high spatial resolution panchromatic image PAN to obtain a first feature map, and extracting a feature map from the low-resolution multispectral image LRMS to obtain a second feature map;

[0011] S2: wavelet decomposing and weighted fusing the first feature map and the second feature map to obtain a multi-channel feature map, and dynamically fusing the multi-channel feature map through a cross-attention gate mechanism to obtain a fused feature map;

[0012] S3: performing residual connection on the fused feature map and the low-resolution multispectral image LRMS, and then performing feature reconstruction to obtain a high-resolution multispectral remote sensing image HRMS.

[0013] Further, the structure of the dynamic wavelet attention fusion module comprises a wavelet decomposition module, a low-frequency fusion module, a high-frequency enhancement module and a Mamba network, wherein the low-frequency fusion module adopts an SE-Net.

[0014] Further, the extracted feature maps are weighted fused by using the dynamic wavelet attention fusion module, and the fusion process comprises:

[0015] wavelet decomposing the extracted first feature map to obtain a PAN low-frequency subband and a PAN high-frequency subband, and respectively wavelet decomposing the extracted second feature map to obtain a LRMS low-frequency subband and a LRMS high-frequency subband;

[0016] weighted fusing the LRMS low-frequency subband and the PAN low-frequency subband by using an SE-Net to obtain a low-frequency subband fused feature map LL fused ;

[0017] The PAN high-frequency sub-band and the LRMS high-frequency sub-band are weighted and fused to obtain a high-frequency fusion result;

[0018] The low-frequency sub-band fusion feature map LL fused is spliced with the high-frequency fusion result to obtain a multi-channel feature map;

[0019] The multi-channel feature map is converted into a Mamba sequence F spectral , and the multi-channel feature map is subjected to windowed Mamba processing to extract a spatial detail feature F spatial .

[0020] The Mamba sequence F spectral and the spatial detail feature F spatial are dynamically fused through a cross-attention gating mechanism to obtain a weighted fusion feature map.

[0021] Further, a fusion weight G is calculated according to the Mamba sequence and the spatial detail feature, and the Mamba sequence and the spatial detail feature are weighted and fused through the cross-attention gating mechanism based on the fusion weight G to obtain the weighted fusion feature map.

[0022] Further, the training phase of the remote sensing image panchromatic sharpening network is divided into three training phases, specifically:

[0023] In the first training phase, the remote sensing image panchromatic sharpening network is supervised trained by using remote sensing image data with reduced resolution;

[0024] In the second training phase, the remote sensing image panchromatic sharpening network is unsupervised alignment trained by using full-resolution remote sensing image data;

[0025] In the third training phase, the remote sensing image panchromatic sharpening network is trained by using mixed remote sensing image data with reduced resolution and full-resolution remote sensing image data.

[0026] In a second aspect, the present application proposes a remote sensing image panchromatic sharpening system for implementing the panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network as described in the first aspect of the present application, which comprises:

[0027] A remote sensing image acquisition module is configured to acquire a low-resolution multispectral image LRMS and a high spatial resolution panchromatic image PAN;

[0028] A feature extraction module is configured to extract feature maps of the high spatial resolution panchromatic image PAN and the low-resolution multispectral image LRMS to obtain a first feature map and a second feature map, respectively;

[0029] a dynamic wavelet attention fusion module, configured to perform wavelet decomposition on the first feature map and the second feature map respectively to obtain low-frequency subbands and high-frequency subbands of each feature map, and obtain a fused feature after weighted fusion;

[0030] a feature reconstruction module, configured to perform feature reconstruction on the fused feature to obtain a high-resolution multispectral image (HRMS).

[0031] The present application has the following advantages:

[0032] 1. The present application designs a panchromatic sharpening network for remote sensing images, wherein the dynamic wavelet attention fusion module utilizes wavelet domain decomposition and cross-modal interaction to achieve adaptive fusion of spectral-spatial information, overcoming the problem of high-frequency blur caused by fixed weights in traditional wavelet methods.

[0033] 3. The present application designs a panchromatic sharpening network for remote sensing images, which balances spectral fidelity and spatial details through frequency enhancement in the wavelet domain and long-range modeling of the Mamba network. To optimize cross-modal deep fusion, the Mamba network is used instead of the traditional Transformer, which reduces the training difficulty of the panchromatic sharpening network for remote sensing images through the linear complexity characteristics of Mamba, breaking through the robustness bottleneck of traditional lightweight models in complex scenarios.

[0034] 3. The present application uses a semi-supervised adversarial learning method to train the panchromatic sharpening network for remote sensing images, combining the stability of supervised learning with the generalization ability of unsupervised learning, and reducing the dependence on full-resolution labels through phased training to alleviate the domain shift problem.

[0035] In summary, the present application systematically solves the problems of spectral distortion, low computational efficiency, and dynamic scene artifacts in the field of panchromatic sharpening, generating high-quality high-resolution multispectral remote sensing images (HRMS). BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 Fig. 1 is a network structure diagram of the panchromatic sharpening network for remote sensing images in an embodiment of the present application;

[0037] Figure 2 Fig. 2 is a step flowchart of generating a high-resolution multispectral remote sensing image (HRMS) using the panchromatic sharpening network for remote sensing images in an embodiment of the present application;

[0038] Figure 3 Fig. 3 is a network structure diagram of the SENet in an embodiment of the present application;

[0039] Figure 4 Fig. 4 is a network structure diagram of the Mamba network in an embodiment of the present application;

[0040] Figure 5A comparison chart of simulation verification results of remote sensing images respectively subjected to panchromatic sharpening by the present application and existing algorithms;

[0041] Figure 6 A system structure schematic diagram of the remote sensing image panchromatic sharpening system in the embodiment of the present application. DETAILED DESCRIPTION

[0042] The terms "first", "second", "third", "fourth" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the terms thus used can be interchanged under appropriate circumstances, and are merely a distinguishing way adopted in the description of the embodiments of the present application for the objects of the same attribute in the description.

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0044] The embodiment of the present application proposes a panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network, which comprises the following steps: collecting a remote sensing image set, respectively acquiring a low-resolution multispectral image LRMS and a high spatial resolution panchromatic image PAN;

[0045] The collected low-resolution multispectral image LRMS and high spatial resolution panchromatic image PAN are input into the trained remote sensing image panchromatic sharpening network, and a high-resolution multispectral remote sensing image HRMS is output.

[0046] Figure 1 A network structure schematic diagram of the remote sensing image panchromatic sharpening network in the embodiment of the present application. Figure 1 The network structure of the remote sensing image panchromatic sharpening network comprises a feature extraction module, a dynamic wavelet attention fusion module and a feature reconstruction module. The feature reconstruction module adopts a generative adversarial network. The input of the network model is a high spatial resolution panchromatic image PAN and a low-resolution multispectral image LRMS. In the model, Conv represents convolution, LeakyReLU represents a LeakyReLU activation function, SENet (Squeeze-and-Excitation Networks) is a convolutional neural network model for image classification, Mamba represents a linear time series network (model) based on a selection state space, and U-net represents a U-shaped network. The output of the model is a high-resolution multispectral remote sensing image HRMS.

[0047] In the illustrated embodiment, with reference to Figure 1 As shown, the network structure of the remote sensing image panchromatic sharpening network includes a feature extraction module, a dynamic wavelet attention fusion module, and a feature reconstruction module. The feature reconstruction module adopts a generative adversarial network.

[0048] Figure 2 A step flowchart for generating a high-resolution multispectral remote sensing image HRMS using the remote sensing image panchromatic sharpening network in the embodiment of the present application.

[0049] In the illustrated embodiment, with reference to Figure 2 As shown, the high-resolution multispectral remote sensing image HRMS is generated using the trained remote sensing image panchromatic sharpening network. The processing process includes:

[0050] S1: Obtain a low-resolution multispectral image LRMS and a high spatial resolution panchromatic image PAN, respectively.

[0051] Specifically, the sensor carried by a satellite, an airplane, or the like is used to remotely detect the earth's surface and the atmosphere layer to collect a remote sensing image set. For example, the remote sensing image set is collected by the fourth-generation high-resolution commercial remote sensing satellite WorldView-III of the United States, or the remote sensing image set is collected by the Chinese civilian optical remote sensing satellite GaoFen-2 (GaoFen-2). Different sensors of the remote sensing satellite can collect remote sensing images of the same target at the same time, and thus the low-resolution multispectral image LRMS and the high spatial resolution panchromatic image PAN of the same target can be obtained.

[0052] S2: Extract a feature map from the high spatial resolution panchromatic image PAN to obtain a first feature map; and extract a feature map from the low-resolution multispectral image LRMS to obtain a second feature map.

[0053] Specifically, a residual dense block is used to extract a feature from the high spatial resolution panchromatic image PAN to obtain the first feature map. The network structure of the residual dense block includes a convolution layer (Conv) and an activation layer (Leaky ReLU activation function). The low-resolution multispectral image LRMS is upsampled to obtain the second feature map.

[0054] S3: Perform wavelet decomposition and weighted fusion on the first feature map and the second feature map to obtain a multi-channel feature map, and perform dynamic fusion on the multi-channel feature map through a cross-attention gate mechanism to obtain a fused feature map.

[0055] In the illustrated embodiment, with reference to Figure 2As shown, the embodiment of the present application designs a remote sensing image panchromatic sharpening network based on cross-resolution adversarial learning and Mamba network, which comprises a feature extraction module, a dynamic wavelet attention fusion module and a feature reconstruction module. The structure of the dynamic wavelet attention fusion module comprises a wavelet decomposition module, a low-frequency fusion module, a high-frequency enhancement module and a Mamba network, wherein the low-frequency fusion module adopts SENet.

[0056] Specifically, the wavelet decomposition module is used to perform wavelet decomposition on the input feature map to extract high-resolution features and low-resolution features, the low-frequency fusion module (specifically, SENet) is used to perform low-frequency fusion on the low-resolution features to obtain a low-frequency fusion feature map, the high-resolution features are weighted and fused to obtain a high-frequency fusion result, and the Mamba network is used to perform cross-modal interactive fusion on the high-frequency fusion result and the low-frequency fusion feature to obtain a multi-channel feature map.

[0057] It should be noted that wavelet decomposition refers to Discrete Wavelet Transform (DWT). MATLAB software can be used to perform wavelet decomposition on the input (feature map) to extract coarse scale (low-resolution) features and fine scale (high-resolution) features of the input (feature map). The coarse scale (low-resolution) features refer to the low-frequency part of the input (feature map), which captures the main and broad features. The fine scale (high-resolution) features refer to the high-frequency part of the input (feature map), which captures the details and rapid changes.

[0058] It should be noted that SENet (Squeeze-and-Excitation Networks) is a convolutional neural network model for image classification, which explicitly models the dependence between channels by adaptively adjusting the feature map weights of each channel, so that the model can pay more attention to feature channels with more information and suppress unimportant feature channels.

[0059] It should be noted that Mamba (Linear-Time Sequence Modeling With Selective State Space) network is a new linear time series network model based on selective state space model (SSM). It combines the characteristics of recurrent neural network (RNN) and convolutional neural network (CNN), and realizes linear or nearly linear expansion of computational cost and sequence length through recursive or convolution operation, thereby significantly reducing the computational complexity.

[0060] Figure 3 The network structure diagram of the SENet in the embodiment of the present application. Figure 3In the specific embodiments, Inception represents interpolation, in the specific embodiments, bicubic interpolation is used for up-sampling, Global Pooling represents global pooling, FC represents a fully connected layer, ReLU represents a ReLU activation function, Sigmoid represents a Sigmoid activation function, and Scale represents a channel weight scaling operation, which is to multiply the channel attention weight (with a value range of [0, 1]) output by the Sigmoid activation function with the original input feature map channel by channel.

[0061] In the illustrated embodiments, reference is made to Figure 3 As shown, the network structure of the SENet includes an input layer, an Inception layer, a pooling layer, a first fully connected layer (ReLU activation function), a second fully connected layer (Sigmoid activation function), a Scale layer, and an output layer.

[0062] The Inception layer represents interpolation of an image. In the specific embodiments, bicubic interpolation is used for up-sampling.

[0063] The FC layer is a fully connected layer. In the specific embodiments, the first fully connected layer (FC) reduces the number of channels from C to C / r (r is a compression ratio, in the specific embodiments, r = 4), thereby reducing the amount of calculation through dimension reduction, and the second fully connected layer restores the number of channels from C / r to C, thereby preparing for subsequent generation of channel attention weights.

[0064] The Scale layer represents a channel weight scaling operation, which is to multiply the channel attention weight (with a value range of [0, 1]) output by the Sigmoid activation function with the original input feature map channel by channel.

[0065] Figure 4 FIG. 2 is a schematic diagram of the network structure of the Mamba network in the specific embodiments of the present application. Figure 4 In the specific embodiments, the network structure of the Mamba network includes a layer normalization layer (Norm), the output of the layer normalization layer (Norm) is connected to two branches, one branch includes a first multilayer perception (MLP), a convolution layer (Conv1d), and a state space model (SSM), the other branch includes a second multilayer perception (MLP) and an activation layer (ReLU activation function), the output results of the two branches are multiplied and input into a third multilayer perception (MLP), the output result of the third multilayer perception (MLP) is connected in residual with the original input to obtain the final output of the Mamba network.

[0066] In the illustrated embodiments, the extracted feature maps are weighted and fused by using the dynamic wavelet attention fusion module, and the fusion process includes:

[0067] S301: Perform wavelet decomposition on the extracted first feature map to obtain the PAN low-frequency sub-band and the PAN high-frequency sub-band; perform wavelet decomposition on the extracted second feature map to obtain the LRMS low-frequency sub-band and the LRMS high-frequency sub-band.

[0068] Specifically, the first feature map is decomposed using the Haar wavelet basis to obtain the LRMS low-frequency subband (LL). LRMS ), LRMS high-frequency subband (LH) LRMS / HL LRMS / HH LRMS LH LRMS HL represents the low-frequency subband along the horizontal direction of an LRMS image. PAN HH represents the low-frequency subband of the LRMS image along the vertical direction. PAN This represents the low-frequency subband along the diagonal direction of the LRMS image.

[0069] The second feature map is decomposed using the Haar wavelet basis to obtain the PAN low-frequency subband (LL). PAN ) and PAN high-frequency subband (LH PAN / HL PAN / HH PAN LH PAN HL represents the high-frequency subband of the PAN image along the horizontal direction. PAN HH represents the high-frequency subband of the PAN image along the vertical direction. PAN This represents the high-frequency subband of the PAN image along the diagonal direction.

[0070] S302: Use SENet to perform weighted fusion of the LRMS low-frequency subband and the PAN low-frequency subband to obtain the low-frequency subband fusion feature map LL. fused .

[0071] Specifically, refer to Figure 2 , Figure 3 As shown, the LRMS low-frequency subband and PAN low-frequency subband are input into SENet to obtain their corresponding weight values. Based on the obtained weights, the LRMS low-frequency subband and PAN low-frequency subband are weighted and fused to output the low-frequency subband fusion feature map LL. fused .

[0072] The low-frequency subband fusion feature map LL fused Represented as:

[0073] LL fused =SENet(LL LRMS )·LL LRMS +(1-SENet(LL PAN ))·LL PAN

[0074] where LL fused denotes the low-frequency subband fusion feature map, SENet(.) denotes the SENet network, LL LRMS denotes the LRMS low-frequency subband, LL PAN denotes the PAN low-frequency subband.

[0075] The low-frequency subband fusion feature map LLfused generated by the SENet channel attention weighted fusion can adaptively allocate the LRMS and PAN low-frequency subband weights according to the channel importance, realize adaptive fusion of spectral-spatial information, effectively balance spectral fidelity and spatial details, and avoid the spectral distortion or insufficient details caused by the traditional fixed weight fusion.

[0076] S303: Weighted fusion is performed on the PAN high-frequency subband and the LRMS high-frequency subband to obtain a high-frequency fusion result.

[0077] Specifically, the gradient amplitude of the PAN image is calculated and used as a complexity index, and based on the complexity index, the PAN high-frequency subband and the LRMS high-frequency subband are weighted and fused by using a convolutional network to obtain a high-frequency fusion result.

[0078] The gradient amplitude of the PAN image is calculated as a complexity index, and the calculation formula is:

[0079]

[0080] where G denotes the complexity index, denotes the high-frequency subband (LH PAN / HL PAN / HH PAN ) of the PAN image in the horizontal direction, denotes the high-frequency subband (LH PAN / HL PAN / HH PAN ) of the PAN image in the vertical direction.

[0081] The complexity index G is used as the input of the subsequent lightweight convolutional network to obtain the weight of each high-frequency subband.

[0082] The weight of each high-frequency subband is learned by a lightweight convolutional network (for example, a simple convolutional network with two layers of 1×1 convolution and ReLU activation function), and the high-frequency subbands are weighted and fused:

[0083] w LH =ConvBlock(G),

[0084] LH fused=w LH ·LH PAN ;

[0085] In the formula, w LH This indicates that the LH obtained through lightweight network learning PAN The weights of the subbands, ConvBlock(.) denotes a lightweight convolutional network, LH fused The LH represents the high-frequency subband of the PAN image along the horizontal direction. PAN The weighted result, LH PAN This represents the high-frequency subband of the PAN image along the horizontal direction.

[0086] w HL =ConvBlock(G),

[0087] HL fused =w HL ·HL PAN ;

[0088] In the formula, w HL This indicates that HL learned through a lightweight network PAN The weights of the subbands, ConvBlock(.) represents a lightweight convolutional network, HL fused HL represents the high-frequency subband of the PAN image along the horizontal direction. PAN The weighted result, HL PAN This represents the high-frequency subband of the PAN image along the vertical direction.

[0089] w HH =ConvBlock(G),

[0090] HH fused =w HH ·HH PAN

[0091] In the formula, w HH This indicates that the HH obtained through lightweight network learning PAN The weights of the subbands, ConvBlock(.) denotes a lightweight convolutional network, HH fused HH represents the high-frequency subband along the diagonal direction of the PAN image. PAN The weighted result, HH PAN This represents the high-frequency subband of the PAN image along the diagonal direction.

[0092] In step S303, the high-frequency subbands are fused by dynamically learning weights based on the PAN gradient magnitude. The fusion intensity can be adaptively adjusted according to the image complexity to optimize the clarity of spatial details (edges, textures), avoid blurring or noise amplification, and enhance cross-modal high-frequency feature interaction, reduce spectral distortion, and improve structural consistency.

[0093] S304: concatenating the low-frequency fusion result LL fused and the high-frequency fusion result (LH fused , HL fused , HH fused ) into a multi-channel feature map.

[0094] In step S304, the low-frequency fusion result and the high-frequency fusion result are concatenated into a multi-channel feature map, which can integrate low-frequency global spectral-spatial fusion information and high-frequency local detail information to form a multi-scale feature representation, provide rich input for subsequent Mamba modeling, facilitate cross-modal deep interaction and dynamic fusion, enhance feature expression capability, help more comprehensively capture image spectral correlation and spatial structure, and improve the balance performance of spectral fidelity and spatial resolution of the fused image.

[0095] S305: converting the multi-channel feature map into a Mamba sequence F spectral , and performing windowed Mamba processing on the multi-channel feature map to extract spatial detail features F spatial .

[0096] Specifically, the multi-spectral channel map is regarded as a sequence, and Mamba sequence modeling is performed along the multi-spectral channel dimension to capture long-range dependencies between wavebands, obtaining a Mamba sequence F spectral .

[0097] The Mamba sequence F spectral is specifically represented as:

[0098] F spectral = Mamba(LL fused , axis = C)

[0099] In the formula, F spectral represents the Mamba sequence, i.e., the result of Mamba sequence modeling on the low-frequency sub-band fusion feature map along the multi-spectral channel dimension, Mamba(.) represents the Mamba network, LL fused represents the low-frequency sub-band fusion feature map, axis = C represents processing along the channel dimension, and C represents the channel dimension.

[0100] In this step, by regarding the multi-spectral channel as a sequence and performing Mamba sequence modeling along the channel dimension to capture spectral long-range dependencies, deep correlations between multi-spectral wavebands can be mined, avoiding the problem of insufficient modeling of spectral correlation in traditional methods, thereby improving the spectral consistency of the fused image, enhancing the global modeling capability of the model for complex spectral features, and finally achieving more accurate spectral fidelity effect.

[0101] Perform 8x8 windowed Mamba processing on the high-frequency sub-band in the multi-channel feature map to extract spatial detail features.

[0102] The spatial detail feature is specifically represented as:

[0103] F spatial = WindowMamba(Concat(LH fused , HL fused , HH fused ))

[0104] In the formula, F spatial represents the spatial detail feature of the high-frequency subband after windowed Mamba processing, WindowMamba(.) represents the windowed Mamba operation, Concat(.) represents the concatenation of feature maps along the channel dimension, LH PAN represents the high-frequency subband of the PAN image along the horizontal direction, HL PAN represents the high-frequency subband of the PAN image along the vertical direction, and HH PAN represents the high-frequency subband of the PAN image along the diagonal direction.

[0105] The windowed Mamba operation processes the concatenated high-frequency subband feature set with a set window size (for example, the window size is set to 8x8). By performing Mamba operations within each window, the Mamba model can focus on local areas and effectively extract spatial detail features in these high-frequency subbands.

[0106] S306: Dynamically fuse the Mamba sequence F spectral and the spatial detail feature F spatial to obtain the weighted fused fusion feature map.

[0107] Specifically, according to the Mamba sequence F spectral and the spatial detail feature F spatial , the fusion weight G is calculated, and the calculation formula is:

[0108] G = Sigmoid(W g · [F spectral , F spatial ])

[0109] Where G represents the calculated fusion weight, Sigmoid(.) represents the Sigmoid activation function, W g is a learnable parameter matrix, F spectral represents the Mamba sequence F spectral , i.e., the result of modeling the low-frequency fusion feature along the multi-spectral channel dimension, and F spatial represents the spatial detail feature, i.e., the result of windowed Mamba processing on the high-frequency subband.

[0110] Based on the fusion weight G, the Mamba sequence F spectral is dynamically fused with the spatial detail feature F spatial through the cross-attention gating mechanism, and a weighted fusion feature map is obtained.

[0111] F fusion = G o F spectral + (1-G) o F spatial

[0112] wherein F fusion represents the weighted fusion feature map, i.e., the Mamba sequence F spectral is dynamically fused with the spatial detail feature F spatial through the cross-attention gating mechanism, and o represents element-wise multiplication.

[0113] S4: After residual connection of the fusion feature map and the low-resolution multispectral image LRMS, feature reconstruction is performed to obtain a high-resolution multispectral remote sensing image HRMS.

[0114] Specifically, the fusion feature map F fusion is wavelet reconstructed and restored to a spatial domain image, which is specifically represented as:

[0115] HRMS base = IDWT(F fusion )

[0116] wherein HRMS base represents the fusion result F fusion restored to a spatial domain image through wavelet reconstruction, and IDWT(·) represents inverse wavelet transform, F fusion represents the weighted fusion feature map, i.e., the Mamba sequence F spectral is dynamically fused with the spatial detail feature F spatial through the cross-attention gating mechanism.

[0117] The spatial domain image HRMS base and the original low-resolution multispectral image LRMS are input into the trained generator after residual connection, feature reconstruction is performed, and a high-resolution multispectral remote sensing image HRMS is obtained, which is specifically represented as:

[0118] HRMS final = HRMS base + UpSample(LRMS)

[0119] wherein HRMS final represents the high-resolution multispectral remote sensing image HRMS, i.e., the fusion result finally output by the model, and UpSample(·) represents up-sampling.

[0120] The fusion feature map and the LRMS image are connected in a residual connection manner to avoid information loss and enhance spectral fidelity of the generated image.

[0121] In the illustrated embodiment, during the entire network (i.e., the remote sensing image panchromatic sharpening network) training stage, the embodiment of the application adopts a three-stage training, and the training process includes:

[0122] The first training stage: using the remote sensing image data with reduced resolution, the entire network is supervised training. In the first training stage, the optimizer adopts Adam (lr=3e-4, β1=0.5, β2=0.999). lr represents the learning rate, e represents the symbol of scientific notation, and 3e-4 is equivalent to 3×10 -4 β1 represents the exponential decay rate of the first moment estimate, and β2 represents the exponential decay rate of the second moment estimate.

[0123] The loss function of the first training stage is designed as follows:

[0124] (1) Joint adversarial loss The idea of generative adversarial network is adopted, the generator tries to generate realistic fusion images, and the discriminator judges whether the generated image is real or generated. The joint adversarial loss The specific calculation formula is:

[0125]

[0126] Wherein, x represents the real reduced resolution fusion image, z represents the noise input to the generator, D(x) represents the output of the discriminator to the real image, D(G(z)) represents the output of the discriminator to the generated image, p real represents the real data distribution, and p fake represents the generated data distribution.

[0127] Through adversarial training, the generator is prompted to generate a more realistic fusion image.

[0128] Spectral angle loss Used to measure the similarity of the generated fusion image and the real high-resolution multispectral image (HRMS) in spectral features. The calculation formula of the spectral angle loss is as follows:

[0129]

[0130] Wherein, N is the total number of image pixels, F i is the spectral vector of the generated image at the i-th pixel point, and Y iis the spectral vector of the real HRMS image at the i-th pixel point. The smaller the spectral angle loss is, the more similar the spectral characteristics of the generated image are to the real image, which helps to ensure the spectral fidelity of the fused image.

[0131] Wavelet high-frequency L1 loss The image is decomposed into subbands of different frequencies using wavelet transform, and L1 loss calculation is performed on the high-frequency subbands.

[0132] Specifically, the generated fused image and the real HRMS image are first wavelet transformed to obtain high-frequency subbands, and then the L1 distance between them is calculated. The wavelet high-frequency L1 loss The specific calculation formula is:

[0133]

[0134] Where M is the total number of all high-frequency subband pixel points, F j (m,n) is the pixel value of the j-th high-frequency subband of the generated image at position (m,n), Y j (m,n) is the pixel value of the j-th high-frequency subband of the real HRMS image at position (m,n). This loss term can help the model better preserve the high-frequency detail information of the image.

[0135] The overall loss function of the first training stage is specifically:

[0136]

[0137] Where, represents the overall loss of the first training stage, represents the overall loss of the first training stage, represents the joint adversarial loss, is the spectral angle loss, is the wavelet high-frequency L1 loss, λ adv represents the first hyperparameter, λ sam represents the second hyperparameter, λ high represents the third hyperparameter, which is used to balance the importance of different loss terms.

[0138] At the beginning of the first training stage, the weight of λ sam can be appropriately increased, so that the entire network model pays more attention to the learning of spectral fidelity; as the training progresses, the weights of λ adv and λ high are gradually adjusted, so that the entire network model can better learn the generation of adversarial and high-frequency details while ensuring spectral fidelity.

[0139] Second training stage: The entire network is trained in an unsupervised manner using full-resolution remote sensing image data. The loss function design of the second training stage is as follows:

[0140] (1) Translation-invariant spectral loss: In order to avoid the problem of traditional spectral loss when processing misaligned regions, a translation-invariant calculation method is adopted. A spectral similarity measure based on sliding window is constructed. A fixed-size window (3x3 window is used in the examples of the present application) is slid on the generated image and the real full-resolution image, and the similarity of the spectral vectors in each window is calculated. Assuming that the window size is (sxs) and the position on the image is (i,j), the translation-invariant spectral loss calculation formula is:

[0141]

[0142] wherein, represents the translation-invariant spectral loss, P is the total number of sliding windows, (i,j) represents the starting position coordinates of the sliding window on the image, (k,l) represents the translation amount in the window, window refers to the sliding window, which is a fixed-size region, F i+m,j+n represents the spectral vector of the pixel at position (m,n) in the window with (i,j) as the starting position in the generated image. Y i+k+m,j+l+n represents the spectral vector of the pixel at position (m,n) in the window after shifting (k,l) from the (i,j) position in the real full-resolution image.

[0143] By calculating the minimum spectral angle under all possible translations, the translation-invariant spectral loss is obtained, which effectively suppresses the artifacts in the misaligned regions.

[0144] (2) Cross-modal contrastive loss: The cross-modal contrastive loss aims to enhance the alignment between spectral features and spatial features. The spectral features and spatial features of the generated image are regarded as two modalities, and positive sample pairs and negative sample pairs are constructed. The spectral and spatial features at the same position are regarded as positive sample pairs, while the spectral and spatial features at different positions are regarded as negative sample pairs. The calculation formula of the cross-modal contrastive loss is:

[0145]

[0146] wherein, Q represents the total number of sample pairs, s q represents the spectral feature vector in the qth positive sample pair, t q represents the spatial feature vector in the qth positive sample pair, and τ is a hyperparameter for adjusting the strength of the contrastive loss. By maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs, the model is prompted to better fuse spectral and spatial features.

[0147] The calculation formula of the overall loss function of the second training stage is specifically:

[0148]

[0149] wherein, denotes the overall loss of the second training stage, denotes the translation-invariant spectral loss, denotes the cross-modal contrastive loss, λ spectral denotes the first weight coefficient, λ contrast denotes the second weight coefficient, λ spectral ∈[0,1], λ contrast ∈[0,1], and λ spectral +λ contrast =1.

[0150] By introducing the translation-invariant spectral loss and the cross-modal contrastive loss to construct the overall loss function of the second training stage, the artifacts of the misaligned regions can be suppressed during the second stage training of the entire network model, so that the entire network model can better adapt to the full-resolution remote sensing image data in real scenes.

[0151] Third training stage: The entire network is trained using mixed low-resolution remote sensing image data and full-resolution remote sensing image data. During training, each batch contains 8 pairs of low-resolution remote sensing image data and 8 pairs of full-resolution remote sensing image data, and the supervised and unsupervised losses are dynamically balanced.

[0152] During training, the weights of the supervised loss and the unsupervised loss are dynamically adjusted according to the training rounds. A weight adjustment function that changes with the training round t is designed:

[0153] For the supervised loss weight α(t), the calculation formula is:

[0154]

[0155] wherein, α(t) denotes the supervised loss weight, T1 denotes a preset training round turning point, for example, T1=30, γ denotes an adjustment rate parameter, and for example, γ=5.

[0156] In the early stage of training, α(t) is small, and the model focuses more on the learning of the unsupervised loss; as the training rounds increase, α(t) gradually increases, and the weight of the supervised loss increases.

[0157] The unsupervised loss weight is set to 1-α(t).

[0158] In the third training stage, by dynamically adjusting the supervised loss and the unsupervised loss, the entire network model can make full use of the advantages of different types of data during the training process, and balance the stability and generalization of the training.

[0159] Based on the same inventive concept, the embodiment of the present application also proposes a panchromatic sharpening system based on cross-resolution adversarial learning and Mamba network, which has the same or similar technical features as the panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network described above. For the same or similar technical features, the following will not be described in detail.

[0160] Referring to Figure 5 The panchromatic sharpening system based on cross-resolution adversarial learning and Mamba network is used to implement the panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network as described in the first aspect of the present application. The system includes:

[0161] A remote sensing image acquisition module is configured to acquire a low-resolution multispectral image LRMS and a high-spatial-resolution panchromatic image PAN.

[0162] A feature extraction module is configured to extract feature maps of the high-spatial-resolution panchromatic image PAN and the low-resolution multispectral image LRMS, to obtain a first feature map and a second feature map, respectively.

[0163] A dynamic wavelet attention fusion module is configured to perform wavelet decomposition on the first feature map and the second feature map, respectively, to obtain low-frequency subbands and high-frequency subbands of each feature map, and to obtain a fused feature after weighted fusion.

[0164] A feature reconstruction module is configured to perform feature reconstruction on the fused feature to obtain a high-resolution multispectral image HRMS.

[0165] Experimental verification:

[0166] Contrast experiments and ablation experiments are designed to prove the effectiveness of each module in the remote sensing image panchromatic sharpening network and the effectiveness of the entire method. The evaluation indexes include spectral angle mapping SAM, global relative error ERGAS, root mean square error RMSE, spatial correlation coefficient SCC, and quality Q n Ablation experiments are designed under four conditions of no wavelet-Mamba interaction, no dynamic weight scheduling, no cross-modal contrast loss, and complete model, to verify the effectiveness of each module mentioned in the present application. Table 1 shows the results of the contrast experiments of the present experiment. Table 2 shows the results of the ablation experiments of the present experiment.

[0167] Table 1: Results of contrast experiments

[0168]

[0169]

[0170] In Table 1, PNN (Pansharpening by convolutional neural networks) represents that the PNN algorithm is used for panchromatic sharpening of the remote sensing image, PanNet (A deep network architecture for pan-sharpening) represents that the PanNet is used for panchromatic sharpening of the remote sensing image, PanCSC (A Model-Driven Deep Unfolding Method for Pan-sharpening) represents that the PanCSC algorithm is used for panchromatic sharpening of the remote sensing image, and GPPNN (Deep Gradient Projection Networks for Pan-sharpening) represents that the GPPNN algorithm is used for panchromatic sharpening of the remote sensing image.

[0171] Figure 5 The simulation verification result comparison chart of panchromatic sharpening of the remote sensing image by using the present application and the existing algorithm respectively. Figure 5 From Table 1, it can be seen that for the four methods of PNN, PanNet, PanCSC and GPPNN, the overall spectral quality on the fusion area is well maintained, but the spectral details of the edge are not continuous enough, and there is spectral distortion; and for the PanCSC method, there is also a significant artifact phenomenon. In contrast, the present application not only restores better spectral details, but also reconstructs more complete spatial details.

[0172] Table 2 ablation experiment results

[0173]

[0174] In Table 2, A represents no wavelet-Mamba interaction, B represents no dynamic weight scheduling, C represents no cross-modal contrast loss, and D represents a complete remote sensing image panchromatic sharpening network.

[0175] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by programs instructing related hardware, and the programs can be stored in a computer readable storage medium, which can include ROM, RAM, magnetic disk or optical disk, etc.

[0176] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network, characterized in that, include: Collect a set of remote sensing images, and acquire the low-resolution multispectral image LRMS and the high spatial resolution panchromatic image PAN respectively; The acquired low-resolution multispectral image (LRMS) and high spatial resolution panchromatic image (PAN) are input into the trained remote sensing image panchromatic sharpening network to generate a high-resolution multispectral remote sensing image (HRMS). The specific processing steps include: Feature maps are extracted from the high spatial resolution panchromatic image PAN to obtain the first feature map; feature maps are extracted from the low resolution multispectral image LRMS to obtain the second feature map. The extracted feature maps are weighted and fused using a dynamic wavelet attention fusion module to obtain a fused feature map. The weighted fusion process includes: The first feature map is decomposed by wavelet to obtain the PAN low-frequency sub-band and the PAN high-frequency sub-band; the second feature map is decomposed by wavelet to obtain the LRMS low-frequency sub-band and the LRMS high-frequency sub-band. The LRMS low-frequency subband and the PAN low-frequency subband are weighted and fused to obtain a low-frequency subband fusion feature map; The PAN high-frequency subband and the LRMS high-frequency subband are weighted and fused to obtain the high-frequency fusion result; The low-frequency sub-band fusion feature map is stitched together with the high-frequency fusion result to form a multi-channel feature map; The multi-channel feature map is converted into a Mamba sequence, and then the multi-channel feature map is processed by windowed Mamba to extract spatial detail features; By dynamically fusing Mamba sequences and spatial detail features through a cross-attention gating mechanism, a weighted fused feature map is obtained. The fused feature map and the low-resolution multispectral image LRMS are residually connected and then feature reconstructed to obtain the high-resolution multispectral remote sensing image HRMS.

2. The panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network according to claim 1, characterized in that, The network structure of the remote sensing image panchromatic sharpening network includes: a feature extraction module, a dynamic wavelet attention fusion module, and a feature reconstruction module. The feature reconstruction module adopts a generative adversarial network.

3. The panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network according to claim 2, characterized in that, The structure of the dynamic wavelet attention fusion module includes a wavelet decomposition module, a low-frequency fusion module, a high-frequency enhancement module, and a Mamba network, wherein the low-frequency fusion module adopts SENet.

4. The panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network according to claim 1, characterized in that, The Mamba sequence is specifically represented as follows: wherein, denotes the Mamba sequence, (.) denotes the Mamba network, denotes the low-frequency sub-band fused feature map, denotes processing along the channel dimension, denotes the channel dimension.

5. The panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network according to claim 1, characterized in that, The spatial detail features are specifically represented as follows: wherein denotes spatial detail features of the high frequency subband of the PAN image, (.) denotes a windowed Mamba operation, (.) denotes concatenation of feature maps along the channel dimension, denotes a high frequency subband of the PAN image along the horizontal direction, denotes a high frequency subband of the PAN image along the vertical direction, denotes a high frequency subband of the PAN image along the diagonal direction.

6. The panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network according to claim 1, characterized in that, According to the Mamba sequence and the spatial detail feature, a fusion weight is calculated , based on the fusion weight , the Mamba sequence and the spatial detail feature are weighted and fused through a cross-attention gate mechanism to obtain a weighted and fused fusion feature map.

7. The panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network according to claim 1, characterized in that, The training phase of the remote sensing image panchromatic sharpening network is divided into three training phases, specifically: In the first training phase, the remote sensing image panchromatic sharpening network was trained in a supervised manner using down-resolution remote sensing image data. In the second training phase, full-resolution remote sensing image data is used to perform unsupervised alignment training on the remote sensing image panchromatic sharpening network. In the third training phase, a remote sensing image panchromatic sharpening network was trained using a mixture of down-resolution and full-resolution remote sensing image data.

8. A remote sensing image panchromatic sharpening system for implementing the panchromatic sharpening method based on cross-resolution adversarial learning and Mamba network according to claim 1, characterized in that, include: The remote sensing image acquisition module is used to acquire low-resolution multispectral images (LRMS) and high spatial resolution panchromatic images (PAN). The feature extraction module is used to extract feature maps from the high spatial resolution panchromatic image PAN and the low resolution multispectral image LRMS, respectively, to obtain the first feature map and the second feature map; The dynamic wavelet attention fusion module is configured to perform wavelet decomposition on the first feature map and the second feature map respectively to obtain low-frequency subbands and high-frequency subbands of each feature map, and obtain a fused feature after weighted fusion. The feature reconstruction module is configured to perform feature reconstruction on the fused feature to obtain a high-resolution multispectral image (HRMS).

Images