Efficient panchromatic sharpening system and method based on multi-frequency hopping Mama network

Abstract

The invention relates to an efficient panchromatic sharpening system and method based on a multi-frequency hopping Mama network, and belongs to the technical field of image processing and remote sensing. The efficient panchromatic sharpening system based on the multi-frequency hopping Mama network comprises an image preprocessing module, an adaptive Mama module, a cross-domain Mama module, a multi-scale fusion module and an output module, the adaptive Mama module is used for respectively extracting image frequency characteristics under different scales from the low-resolution multispectral image and the high-resolution panchromatic image after the resolution is adjusted to obtain space and spectral information; the cross-domain Mama module is used for realizing cross-domain Mama module space and spectral feature fusion according to the image frequency features under different scales extracted by the adaptive Mama module, and integrating high-frequency details in the panchromatic image and spectral information in the multispectral image; and the multi-scale fusion module is used for further integrating the multi-scale feature images integrated by the cross-domain Mama module. According to the method, the spatial and spectral features can be extracted more efficiently, and the spatial detail representation of the image is enhanced on the premise of maintaining the spectral consistency.

Description

Technical Field

[0001] The invention relates to an efficient full color sharpening system and method based on a multi-frequency hopping Mamba network, belonging to the technical field of image processing and remote sensing. Background Art

[0002] Existing remote sensing imaging technologies usually have limitations when it comes to obtaining high spatial resolution and high spectral resolution images. Although panchromatic images have high spatial resolution, they lack rich spectral information; while multispectral images contain rich spectral information, but their spatial resolution is low. In order to improve both spectral and spatial resolution, image fusion technology is widely used. Existing image fusion technologies mainly include methods based on multi-resolution analysis, model optimization, and component replacement.

[0003] The method based on multi-resolution analysis improves the spatial resolution by extracting high-frequency spatial details from the panchromatic image and fusing them with the spectral information of the multispectral image. The model optimization method solves the image fusion problem as a mathematical optimization task and improves the fusion accuracy through prior information. The component replacement method achieves the optimal fusion of features by constructing a replacement model of spectral and spatial features. However, traditional methods still have certain shortcomings in dealing with complex image structures, global information modeling, and spectral and spatial feature fusion. In recent years, convolutional neural networks (CNNs) have been widely used in image fusion tasks. They have powerful nonlinear fitting capabilities and can effectively improve the fusion effect. However, the local receptive field of CNN limits its global information modeling capabilities. In addition, although the Transformer-based model enhances the global modeling capability, it has high computational complexity and is difficult to process large-scale remote sensing images.

[0004] Therefore, the present invention proposes a multi-frequency hopping Mamba network system combining an adaptive Mamba module (AMM) and a cross-domain Mamba module (CDM), which achieves more efficient fusion of panchromatic images and multispectral images without significantly increasing the computational complexity. Summary of the invention

[0005] The present invention solves the problem of insufficient spatial resolution of multispectral images and missing spectral information of panchromatic images in the prior art, and provides an efficient panchromatic sharpening system and method based on a multi-frequency hopping Mamba network. In this system, by cooperating with an adaptive Mamba module and a cross-domain Mamba module, spatial and spectral features can be extracted more efficiently, and the spatial detail performance of the image can be enhanced while maintaining spectral consistency. The present invention is implemented by the following technical solutions.

[0006] An efficient full color sharpening system based on a multi-frequency hopping Mamba network, comprising an image preprocessing module, an adaptive Mamba module, a cross-domain Mamba module, a multi-scale fusion module and an output module;

[0007] The image preprocessing module is used to adjust the resolution of the low-resolution multispectral image to be consistent with the resolution of the high-resolution panchromatic image;

[0008] The adaptive Mamba module is used to extract image frequency features at different scales from the low-resolution multispectral image and the high-resolution panchromatic image after adjusting the resolution, and obtain spatial and spectral information;

[0009] The cross-domain Mamba module is used to extract the image frequency features at different scales from the adaptive Mamba module to achieve the fusion of the cross-domain Mamba module space and spectral features, and integrate the high-frequency details in the panchromatic image with the spectral information in the multispectral image;

[0010] The multi-scale fusion module is used to further integrate the multi-scale feature images after the cross-domain Mamba module to generate high-resolution multispectral images;

[0011] The output module is used to generate high-resolution multispectral images.

[0012] The adaptive Mamba module includes a multi-layer convolution operation unit, a maximum pooling and average pooling operation unit and an activation function unit.

[0013] The multi-layer convolution operation unit is used to dynamically adjust the convolution kernel size and stride, and can flexibly adjust the receptive field size according to the complexity and feature distribution of the input image. In the high-frequency area of ​​the image, the AMM module uses a larger convolution kernel to capture the global features and high spatial details of the image; in the low-frequency area, the module uses a smaller convolution kernel to extract spectral information in detail.

[0014] The maximum pooling and average pooling operation units are used to ensure that feature information at different scales is fully captured;

[0015] The activation function unit generates weighted coefficients to automatically enhance the feature extraction capability of key areas.

[0016] The adaptive Mamba module has the ability to dynamically adjust the kernel size and stride of the multi-layer convolution operation according to the following formula:

[0017] I c =Max(Conv(I M ))+Mean(Conv(I M ))×Conv(I M ) (1);

[0018] in Conv(·) represents the convolutional layer, Max(·) represents the maximum pooling, Mean(·) represents the average pooling, I M Represents the features of the original input, I c Represents the dynamically adjusted features, and the dimension of each generated feature matrix is ​​H×W×C; the dynamic characteristics of the adaptive Mamba module enable it to dynamically adjust the internal state according to different inputs, thereby enhancing the model's sensitivity to input changes and adapting to complex sequence modeling tasks.

[0019] The cross-domain Mamba module includes a cross-domain convolution operation unit and a frequency domain processing unit.

[0020] The cross-domain convolution operation unit is used to adjust the size of the convolution kernel, project the image frequency features at different scales into the frequency domain space, and perform cross-domain convolution fusion; the frequency domain processing unit is used to capture the details of spectral changes, and combined with the convolution operation, ensure the consistency of spectral information and spatial details at multiple scales. By adjusting the size of the convolution kernel, the CDM module can flexibly adapt to image details of different frequencies, ensuring that the generated high-resolution images have excellent performance in both spectral fidelity and spatial resolution.

[0021] The cross-domain Mamba module outputs a multi-scale feature image represented as:

[0022] Mamba(I fusion )=MLP(I PAN +I MS ) (2)

[0023] Among them, I fusion is the fused multi-scale feature image, represents a high-resolution full-color image (Pan image), Represents a low-resolution multispectral image (MS image).

[0024] The cross-domain convolution operation unit in the cross-domain Mamba module includes a 3×3 convolution operation, which further fuses features of different resolutions, expressed as:

[0025] ADA( Conv (I S ))=Avg(Max(Conv(I fusion )))+Fourier(Conv(I fusion )) (3)

[0026] Among them I S represents multi-scale fusion features, Fourier represents Fourier frequency domain enhancement, Avg represents average pooling, Max represents maximum pooling, Conv(·) represents convolutional layer, Ifusion It is the fused multi-scale feature images with different resolutions.

[0027] The multi-scale fusion module includes a multi-scale fusion strategy unit and a residual connection technology unit, which ensures that the input original image information is fully preserved during the fusion process and improves the overall image quality through layer-by-layer feature integration.

[0028] An efficient full color sharpening method based on a multi-frequency hopping Mamba network comprises the following steps:

[0029] S1, the image preprocessing module uses the interpolation upsampling method to adjust the resolution of the low-resolution multispectral image to be consistent with the resolution of the high-resolution panchromatic image;

[0030] S2, a method of extracting image frequency features at different scales from low-resolution multispectral images and high-resolution panchromatic images after adjusting the resolution using an adaptive Mamba module;

[0031] S3, a method of using a cross-domain Mamba module to extract image frequency features at different scales to achieve spatial and spectral feature fusion;

[0032] S4, using a multi-scale fusion module to further fuse the fused multi-scale feature images to generate a high-resolution multispectral image;

[0033] S5. A method of outputting the generated high-resolution multispectral image using an output module.

[0034] The S4 high-resolution multispectral image is optimized using the L1 loss function and the ERGAS loss function to generate an optimized high-resolution multispectral image; the optimized high-resolution multispectral image is output through an output module.

[0035] The beneficial effects of the present invention are:

[0036] 1. Efficient fusion of global and local features: Through the adaptive Mamba module, the system can flexibly extract multi-scale features of the image, ensuring that spatial and spectral features can be fully extracted in different regions; the cross-domain Mamba module further improves the overall performance of the image by fusing frequency domain and spatial domain information.

[0037] 2. Efficient feature extraction and fusion mechanism: By dynamically adjusting the convolution kernel and cross-domain convolution operations, the system can achieve efficient feature extraction and fusion without significantly increasing the computational complexity, making it suitable for real-time processing of large-scale remote sensing images.

[0038] 3. Significant improvement in image quality: While maintaining spectral consistency, the present invention significantly enhances the retention of spatial details of the image, and is suitable for use in scenarios with high requirements for image details, such as urban monitoring and agricultural mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the architecture of the efficient full color sharpening system based on the multi-frequency hopping Mamba network of the present invention;

[0040] Figure 2 It is a schematic diagram of the structure of the adaptive Mamba module (AMM) of the present invention, showing the dynamic adjustment process of the convolution kernel in the multi-layer convolution operation unit in the adaptive Mamba module;

[0041] Figure 3 It is a schematic diagram of the structure of the cross-domain Mamba module (CDM) of the present invention, showing the process of feature projection and cross-domain fusion of the cross-domain convolution operation unit in the cross-domain Mamba module (CDM);

[0042] Figure 4 is a high-resolution full-color image (Pan image) of the present invention on the IKONOS data set;

[0043] Figure 5 is a low-resolution multispectral image (MS image) of the present invention on the IKONOS data set;

[0044] Figure 6 The present invention outputs and generates high-resolution multispectral images on the IKONOS dataset. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0046] Example 1

[0047] like Figure 1 As shown, the efficient full color sharpening system based on the multi-frequency jumping Mamba network includes an image preprocessing module, an adaptive Mamba module, a cross-domain Mamba module, a multi-scale fusion module and an output module;

[0048] The image preprocessing module is used to adjust the resolution of the low-resolution multispectral image to be consistent with the resolution of the high-resolution panchromatic image;

[0049] The adaptive Mamba module is used to extract image frequency features at different scales from the low-resolution multispectral image and the high-resolution panchromatic image after adjusting the resolution, and obtain spatial and spectral information;

[0050] The cross-domain Mamba module is used to extract the image frequency features at different scales from the adaptive Mamba module to achieve the fusion of the cross-domain Mamba module space and spectral features, and integrate the high-frequency details in the panchromatic image with the spectral information in the multispectral image;

[0051] The multi-scale fusion module is used to further integrate the multi-scale feature images after the cross-domain Mamba module to generate high-resolution multispectral images;

[0052] The output module is used to generate high-resolution multispectral images.

[0053] from Figure 1 It can also be seen that the model framework includes an Adaptive Mamba Module and a Cross-Domain Mamba. In the Adaptive Mamba Module, the Mamba state space model and channel features are combined to achieve adaptive extraction and fusion of multimodal features, enhancing the global perception and generalization performance of the network. In Cross-DomainMamba, the deep fusion of spatial and spectral features is achieved, and the responsiveness to input features is improved through adaptive convolution kernels, which improves the effectiveness of cross-modal information fusion.

[0054] like Figure 2 As shown in the figure, the adaptive Mamba module includes multi-layer convolution operation units, maximum pooling and average pooling operation units, and activation function units. The multi-layer convolution operation unit is used to dynamically adjust the convolution kernel size and stride, and can flexibly adjust the receptive field size according to the complexity and feature distribution of the input image. In the high-frequency area of ​​the image, the AMM module uses a larger convolution kernel to capture the global features and high spatial details of the image; in the low-frequency area, the module uses a smaller convolution kernel to extract spectral information in detail.

[0055] The maximum pooling and average pooling operation units are used to ensure that feature information at different scales is fully captured.

[0056] The activation function unit generates weighted coefficients to automatically enhance the feature extraction capability of key areas.

[0057] The multi-layer convolution operation in the adaptive Mamba module dynamically adjusts the convolution kernel size and stride according to the following formula: The multi-layer convolution operation in the adaptive Mamba module dynamically adjusts the convolution kernel size and stride according to the following formula:

[0058] I c =Max(Conv(I M ))+Mean(Conv(I M ))×Conv(I M) (1);

[0059] in Conv(·) represents the convolutional layer, Max(·) represents the maximum pooling, Mean(·) represents the average pooling, I M Represents the features of the original input, I c Represents the dynamically adjusted features, and the dimension of each generated feature matrix is ​​H×W×C; the dynamic characteristics of the adaptive Mamba module enable it to dynamically adjust the internal state according to different inputs, thereby enhancing the model's sensitivity to input changes and adapting to complex sequence modeling tasks

[0060] like Figure 3 As shown, the cross-domain Mamba module includes a cross-domain convolution operation unit and a frequency domain processing unit.

[0061] The cross-domain convolution operation unit is used to adjust the size of the convolution kernel, project the image frequency features at different scales into the frequency domain space, and perform cross-domain convolution fusion; the frequency domain processing unit is used to capture the details of spectral changes, and combined with the convolution operation, ensure the consistency of spectral information and spatial details at multiple scales. By adjusting the size of the convolution kernel, the CDM module can flexibly adapt to image details of different frequencies, ensuring that the generated high-resolution images have excellent performance in both spectral fidelity and spatial resolution.

[0062] The cross-domain Mamba module outputs a multi-scale feature image represented as:

[0063] Mamba(I fusion )=MLP(I PAN +I MS ) (2)

[0064] Among them, I fusion is the fused multi-scale feature image, represents a high-resolution full-color image (Pan image), Represents a low-resolution multispectral image (MS image).

[0065] The cross-domain convolution operation unit in the cross-domain Mamba module includes a 3×3 convolution operation, which further fuses features of different resolutions, expressed as:

[0066] ADA( Conv (I S ))=Avg(Max(Conv(I fusion )))+Fourier(Conv(I fusion )) (3)

[0067] Among them I Srepresents multi-scale fusion features, Fourier represents Fourier frequency domain enhancement, Avg represents average pooling, Max represents maximum pooling, Conv(·) represents convolutional layer, I fusion It is the fused multi-scale feature images with different resolutions.

[0068] The multi-scale fusion module includes a multi-scale fusion strategy unit and a residual connection technology unit, which ensures that the input original image information is fully preserved during the fusion process and improves the overall image quality through layer-by-layer feature integration.

[0069] The efficient full color sharpening method based on the multi-frequency jump Mamba network comprises the following steps:

[0070] S1, the image preprocessing module uses the interpolation upsampling method to adjust the resolution of the low-resolution multispectral image to be consistent with the resolution of the high-resolution panchromatic image;

[0071] S2, a method of extracting image frequency features at different scales from low-resolution multispectral images and high-resolution panchromatic images after adjusting the resolution using an adaptive Mamba module;

[0072] S3, a method of using a cross-domain Mamba module to extract image frequency features at different scales to achieve spatial and spectral feature fusion;

[0073] S4, using a multi-scale fusion module to further fuse the fused multi-scale feature images to generate a high-resolution multispectral image;

[0074] S5. A method of outputting the generated high-resolution multispectral image using an output module.

[0075] The high-resolution multispectral image (HRMS) output by the present invention is not only superior to the existing fusion methods in visual effect, but also shows significant improvement in quantitative indicators, such as spectral angle mapping (SAM), global relative error (ERGAS) and spatial correlation coefficient (SCC).

[0076] Experiment and performance verification

[0077] In order to verify the effectiveness of the present invention, experiments were conducted on the IKONOS and WorldView-2 datasets. Figure 4 It is a high-resolution full-color image on the IKONOS dataset used in the experiment. Figure 5It is a low-resolution multispectral image on the IKONOS dataset in the experiment. In the experiment, the system uses the L1 loss function and the ERGAS loss function to optimize the spectral consistency and spatial detail retention of the generated image through the S4 high-resolution multispectral image, and generates an optimized high-resolution multispectral image; the optimized high-resolution multispectral image is output through the output module, such as Figure 6 The experimental results and image comparisons show that the efficient full color sharpening system based on the multi-frequency jump Mamba network proposed in the present invention is superior to the traditional method in multiple quantitative evaluation indicators, and has obvious advantages in image details and spectral fidelity.

[0078] like Figure 6 As shown in the figure, the experimental results clearly show the excellent performance of the HRMS images generated by the system of the present invention in terms of spectral fidelity and spatial details. Compared with the existing image fusion technology, the present invention not only improves the visual quality of the image, but also has obvious advantages in computational efficiency.

[0079] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An efficient panchromatic sharpening system based on a multi-frequency hopping Mamba network, characterized by: It includes image preprocessing module, adaptive Mamba module, cross-domain Mamba module, multi-scale fusion module and output module; The image preprocessing module is used to adjust the resolution of the low-resolution multispectral image to be consistent with the resolution of the high-resolution panchromatic image; The adaptive Mamba module is used to extract image frequency features at different scales from the low-resolution multispectral image and the high-resolution panchromatic image after adjusting the resolution, and obtain spatial and spectral information; The cross-domain Mamba module is used to extract the image frequency features at different scales from the adaptive Mamba module to achieve the fusion of the cross-domain Mamba module space and spectral features, and integrate the high-frequency details in the panchromatic image with the spectral information in the multispectral image; The multi-scale fusion module is used to further integrate the multi-scale feature images after the cross-domain Mamba module to generate high-resolution multispectral images; The output module is used to generate high-resolution multispectral images.

2. The efficient full color sharpening system based on multi-frequency hopping Mamba network according to claim 1, characterized in that: The adaptive Mamba module includes a multi-layer convolution operation unit, a maximum pooling and average pooling operation unit and an activation function unit; the multi-layer convolution operation unit is used to dynamically adjust the convolution kernel size and stride; the maximum pooling and average pooling operation unit is used to ensure that feature information at different scales is fully captured; the activation function unit is used to automatically enhance the feature extraction capability of key areas.

3. The efficient full color sharpening system based on multi-frequency hopping Mamba network according to claim 2, characterized in that: The adaptive Mamba module has the ability to dynamically adjust the kernel size and stride of the multi-layer convolution operation according to the following formula: I c =Max(Conv(I M ))+Mean(Conv(I M ))×Conv(I M ) (1); in Conv(·) represents the convolutional layer, Max(·) represents the maximum pooling, Mean(·) represents the average pooling, I M Represents the features of the original input, I c Represents the dynamically adjusted features, and the dimension of each generated feature matrix is ​​H×W×C.

4. The efficient full color sharpening system based on multi-frequency hopping Mamba network according to claim 1, characterized in that: The cross-domain Mamba module includes a cross-domain convolution operation unit and a frequency domain processing unit. The cross-domain convolution operation unit is used to adjust the size of the convolution kernel, and to project the image frequency features at different scales into the frequency domain space and perform cross-domain convolution fusion; the frequency domain processing unit is used to capture the details of spectral changes.

5. The efficient full color sharpening system based on multi-frequency hopping Mamba network according to claim 4, characterized in that: The cross-domain Mamba module outputs a multi-scale feature image represented as: Mamba(Ifusion)=MLP(I PAN +I MS ) (2) Among them, I fusion is the fused multi-scale feature image, represents a high-resolution full-color image, Represents a low-resolution multispectral image.

6. The efficient full color sharpening system based on multi-frequency hopping Mamba network according to claim 5, characterized in that: The cross-domain convolution operation unit in the cross-domain Mamba module includes a 3×3 convolution operation, which further fuses features of different resolutions, expressed as: ADA( Conv (I S ))=Avg(Max(Conv(I fusion )))+Fourier(Conv(I fusion )) (3) Among them I S represents multi-scale fusion features, Fourier represents Fourier frequency domain enhancement, Avg represents average pooling, Max represents maximum pooling, Conv(·) represents convolutional layer, I fusion It is the fused multi-scale feature images with different resolutions.

7. The efficient full color sharpening system based on multi-frequency hopping Mamba network according to claim 1, characterized in that: The multi-scale fusion module includes a multi-scale fusion strategy unit and a residual connection technology unit.

8. An efficient full color sharpening method based on a multi-frequency hopping Mamba network, characterized by the steps include: S1, the image preprocessing module uses the interpolation upsampling method to adjust the resolution of the low-resolution multispectral image to be consistent with the resolution of the high-resolution panchromatic image; S2, a method of extracting image frequency features at different scales from low-resolution multispectral images and high-resolution panchromatic images after adjusting the resolution using an adaptive Mamba module; S3, a method of using a cross-domain Mamba module to extract image frequency features at different scales to achieve spatial and spectral feature fusion; S4, using a multi-scale fusion module to further fuse the fused multi-scale feature images to generate a high-resolution multispectral image; S5. A method of outputting the generated high-resolution multispectral image using an output module.

9. The efficient full color sharpening method based on multi-frequency hopping Mamba network according to claim 8, characterized in that: The S4 high-resolution multispectral image is optimized using the L1 loss function and the ERGAS loss function to generate an optimized high-resolution multispectral image; the optimized high-resolution multispectral image is output through an output module.

Images