Multispectral image fusion method and system based on spatial spectral difference prior guidance

Abstract

The present invention relates to the technical field of multispectral image fusion, and provides a multispectral image fusion method and system based on spatial-spectral difference prior guidance. The method comprises upsampling an LRMS image and subtracting and splicing it band by band from a panchromatic image to obtain a spatial difference input feature map; extracting a multiscale spatial feature map through a spatial difference prior guidance branch; subtracting the upsampled LRMS image from a degraded panchromatic image band by band, and splicing it with the upsampled LRMS image to obtain a spectral difference input feature map; extracting a multiscale spectral feature map through a spectral difference prior guidance branch; and fusing the multiscale spatial feature map and the multiscale spectral feature map through a spatial-spectral feature fusion branch to obtain an HRMS image. The present invention uses the spectral and spatial difference information of the panchromatic image and the LRMS image at the pixel level as a prior to guide the extraction and fusion of complementary information, thereby achieving accurate high-resolution multispectral image reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of multispectral image fusion, and in particular to a multispectral image fusion method and system based on spatial spectral difference prior guidance. Background Art

[0002] Low-resolution multispectral (LRMS) images possess rich spectral information and can accurately reflect spectral features, but their low spatial resolution makes it difficult to capture fine details. In contrast, panchromatic (PAN) images possess higher spatial resolution and can provide clear spatial structural information, but they lack spectral information. Due to the physical limitations of current satellite sensors' imaging capabilities, it is not yet possible to directly acquire high-resolution multispectral (HRMS) images with both high spatial and spectral resolution. This limitation, to a certain extent, restricts the practical value and effectiveness of remote sensing imagery in key application scenarios such as agricultural monitoring, disaster response, and precision navigation.

[0003] Pansharpening technology aims to fuse a panchromatic image with a LRMS image to generate a HRMS image. Deep learning methods have attracted widespread attention and made significant progress in this task due to their powerful feature representation and nonlinear fitting capabilities. Existing deep learning-based methods directly extract features from the two source images for fusion, but fail to fully consider the differences between spectral and spatial information at the pixel level. This limits the model's ability to extract complementary features, resulting in spectral distortion and loss of spatial texture detail in the fusion results, and the model cannot accurately extract the required features.

[0004] Most existing pan-sharpening methods typically only consider fusing spatial and spectral features at the same scale during the fusion phase, ignoring the complementarity and correlation between features at different scales. This limitation makes it difficult for the model to fully exploit the rich contextual semantics contained in multi-scale information, limiting the performance of the final fused image in terms of spatial detail recovery and spectral fidelity. Existing deep learning-based pan-sharpening methods improve model performance by stacking convolutional layers and Transformer attention layers, but the high model complexity limits its application in real-world scenarios. Summary of the Invention

[0005] The present invention aims to address at least one of the technical problems existing in the related art. To this end, the present invention provides a multispectral image fusion method and system based on spatial-spectral difference prior guidance. By using the spectral and spatial difference information at the pixel level between the panchromatic image and the LRMS image as a prior, the system guides the extraction and fusion of complementary information, achieving accurate high-resolution multispectral image reconstruction.

[0006] The present invention provides a multispectral image fusion method based on spatial spectral difference prior guidance, comprising:

[0007] S1: Acquire LRMS images and panchromatic images through remote sensing optical imaging system;

[0008] S2: Upsample the LRMS image to obtain an upsampled LRMS image of the same size as the panchromatic image, subtract the upsampled LRMS image from the panchromatic image band by band to obtain a spatial difference feature map, and concatenate the spatial difference feature map with the panchromatic image to obtain a spatial difference input feature map;

[0009] S3: Multi-scale feature extraction is performed on the spatial difference input feature map through the spatial difference prior guidance branch to obtain a multi-scale spatial feature map;

[0010] S4: downsample and upsample the panchromatic image to obtain a degraded panchromatic image, subtract the upsampled LRMS image from the degraded panchromatic image band by band to obtain a spectral difference feature map, and concatenate the spectral difference feature map with the upsampled LRMS image to obtain a spectral difference input feature map;

[0011] S5: extract multi-scale features from the spectral difference input feature map through the spectral difference prior guided branch to obtain a multi-scale spectral feature map;

[0012] S6: The multi-scale spatial feature map and the multi-scale spectral feature map are fused through the spatial-spectral feature fusion branch to obtain the HRMS image.

[0013] Furthermore, in step S3, the spatial difference prior-guided branch includes spatial feature extraction layers at multiple scales, the first-scale spatial feature extraction layer includes a 3×3 convolutional layer and a spatial feature extraction block, and the remaining-scale spatial feature extraction layers include downsampling, spatial feature extraction blocks, and prior-guided feature enhancement blocks;

[0014] The spatial difference prior guide branch is used to extract multi-scale features from the spatial difference input feature map, and the calculation expression for the multi-scale spatial feature map is:

[0015]

[0016]

[0017] in, is the first scale space feature map, is the spatial feature extraction block, is a 3×3 convolutional layer, Input feature map for spatial differences, is the full-color image of the first scale, For the Full-scale color image, For the Scale-space feature map, is a priori guided feature enhancement block, is a double downsampling operation, For the Scale-space feature map.

[0018] Furthermore, the spatial feature extraction block includes two convolutional layers, two activation layers and a spatial feature enhancement unit, and the calculation expression is:

[0019]

[0020]

[0021]

[0022] in, is the output feature after two convolutional layers and two activation layers, is the GELU activation function, is a 7×7 convolution, For channel splicing operation, The input features of the spatial feature extraction block, is the spatial weight map, For the Full-scale color image, is a high-pass filter, is the global average pooling, is the global maximum pooling operation, For the The output features of the scale-space feature extraction block, is element-wise multiplication, The output features of the first scale-space feature extraction block.

[0023] Furthermore, in step S5, the spectral difference prior guidance branch includes spectral feature extraction layers of multiple scales, the first scale spectral feature extraction layer includes a 3×3 convolution layer and a spectral feature extraction block, and the remaining scale spectral feature extraction layers include downsampling, spectral feature extraction blocks and prior guidance feature enhancement blocks;

[0024] The spectral difference prior guide branch is used to extract multi-scale features from the spectral difference input feature map, and the calculation expression for the multi-scale spectral feature map is:

[0025]

[0026]

[0027] in, is the first scale spectral feature map, is the spectral feature extraction block, Input feature map for spectral differences, is the upsampled LRMS image, For the Scale spectral feature map, For the Scale spectral feature map.

[0028] Furthermore, the spectral feature extraction block includes a spectral feature enhancement unit and two spectral feature extraction units.

[0029] The calculation expression is:

[0030]

[0031]

[0032]

[0033] in, Output features for the spectral feature extraction unit, is the spectral feature extraction unit, is the input feature of the spectral feature extraction block, is a 7×7 convolution, is the GELU activation function, is a multi-layer perceptron, is the global average pooling operation, Element-wise multiplication, For the The output features of the scale spectrum feature extraction block, The output features of the first-scale spectral feature extraction block.

[0034] Furthermore, the prior-guided feature enhancement block includes:

[0035] The difference input feature map is downsampled and passed through 3 3 convolutional layers to obtain supplementary features;

[0036] The first The output features of the scale feature extraction block are subtracted from the supplementary features to obtain the residual features;

[0037] The residual features are passed through 3 3. Convolutional layer and Sigmoid function processing to obtain the difference weight map;

[0038] The difference weight map and Multiply the output features of the scale feature extraction block to obtain detail enhancement features;

[0039] The detail enhancement features are concatenated with the supplementary features and fused with the detail enhancement features through channel attention and convolution operations to obtain the first scale feature map;

[0040] Among them, in step S3, the difference input feature map is the spatial difference input feature map, The scale feature extraction block is Scale space feature extraction block, The scale feature map is Scale-space feature map;

[0041] In step S5, the difference input feature map is the spectral difference input feature map. The scale feature extraction block is Scale spectrum feature extraction block, The scale feature map is Scale spectral feature map.

[0042] Furthermore, the spatial-spectral feature fusion branch includes a spatial-spectral feature fusion block, which fuses the spatial feature map and the spectral feature map of the same scale;

[0043] The first Scale space feature map and The scale spectral feature maps are spliced ​​to obtain the scale-space spectral feature maps;

[0044] No. The scale space spectral feature map is generated by 3×3 convolution layer, GELU activation function, 3×3 convolution layer and Sigmoid function. Scale-space feature weight map;

[0045] Through the Scale space feature weight map and the The spatial feature map of the scale is multiplied element by element and then Add the scale spectral feature maps to obtain the Scale-space spectral fusion feature map.

[0046] Furthermore, the spatial-spectral feature fusion branch also includes a cross-scale feature fusion block, which realizes the fusion of adjacent scale features by establishing a global correlation between cross-scale features.

[0047] The first The scale space spectrum fusion feature is upsampled to the Scale space spectral fusion features the same size;

[0048] The spatial-spectral fusion features of the two scales are reshaped into tokens and normalized before being fed into the linear cross attention unit and combined with the first Add up the scale tokens to get the Scale Enhancement Token;

[0049] No. After the scale-enhanced tokens are reshaped into feature maps, The scale-space spectral fusion features are added to obtain the cross-scale feature fusion block in the first i Output on scale;

[0050] Among them, the linear cross attention unit is:

[0051] The first The scale-normalized token mapping is the query matrix;

[0052] No. The scale-normalized tokens enter two branches,

[0053] The first branch aggregates local information through a linear layer, a 3×3 convolution, and a SiLU activation function to obtain a value matrix, which is further mapped to a key matrix through a linear layer. The linear attention is calculated using the query matrix, the value matrix, and the key matrix to obtain the linear attention output;

[0054] The second branch passes through the linear layer and SiLU activation mapping, multiplies it with the linear attention output, and passes through a linear layer mapping to obtain the output of the linear cross attention unit.

[0055] Furthermore, the spatial-spectral feature fusion branch also includes a multi-scale feature fusion block, which is used to generate a high-resolution multispectral image.

[0056] The calculation expression is:

[0057]

[0058] in, For high-resolution multispectral images, is a 3×3 convolutional layer, is a 1×1 convolutional layer, is a double upsampling operation, For the cross-scale feature fusion block in Output on scale, .

[0059] The present invention further provides a multispectral image fusion system based on spatial spectral difference prior guidance, which is used for the above-mentioned multispectral image fusion method based on spatial spectral difference prior guidance, comprising:

[0060] An acquisition module, wherein the acquisition module acquires an LRMS image and a panchromatic image through a remote sensing optical imaging system;

[0061] a spatial difference feature construction module, which upsamples the LRMS image to obtain an upsampled LRMS image of the same size as the panchromatic image, subtracts the upsampled LRMS image from the panchromatic image band by band to obtain a spatial difference feature map, and concatenates the spatial difference feature map with the panchromatic image to obtain a spatial difference input feature map;

[0062] A spatial feature extraction module, wherein the spatial feature extraction module performs multi-scale feature extraction on the spatial difference input feature map through a spatial difference prior guided branch to obtain a multi-scale spatial feature map;

[0063] a spectral difference feature construction module, which downsamples and upsamples the panchromatic image to obtain a degraded panchromatic image, subtracts the upsampled LRMS image from the degraded panchromatic image band by band to obtain a spectral difference feature map, and concatenates the spectral difference feature map with the upsampled LRMS image to obtain a spectral difference input feature map;

[0064] A spectral feature extraction module, wherein the spectral feature extraction module performs multi-scale feature extraction on the spectral difference input feature map through a spectral difference prior guided branch to obtain a multi-scale spectral feature map;

[0065] The fusion module fuses the multi-scale spatial feature map and the multi-scale spectral feature map through a spatial-spectral feature fusion branch to obtain an HRMS image.

[0066] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0067] The present invention uses the spectral and spatial difference information of the panchromatic image and the LRMS image at the pixel level as a priori to guide the extraction and fusion of complementary information, significantly improving the accuracy of feature extraction and the fusion effect, and realizing accurate reconstruction of high-resolution multispectral images.

[0068] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 This is a flow chart of a multispectral image fusion method based on spatial spectral difference prior guidance provided by the present invention.

[0071] Figure 2 This is a schematic diagram of a model framework of a multispectral image fusion method based on spatial spectral difference prior guidance provided by the present invention.

[0072] Figure 3 It is a structural diagram of the spatial feature extraction block provided by the present invention.

[0073] Figure 4 It is a structural diagram of the prior-guided feature enhancement block provided by the present invention.

[0074] Figure 5 It is a structural diagram of the spectral feature extraction block provided by the present invention.

[0075] Figure 6 It is a structural schematic diagram of the spatial spectrum feature fusion block provided by the present invention.

[0076] Figure 7 Schematic diagram of the structure of the cross-scale feature fusion block provided by the present invention.

[0077] Figure 8 Schematic diagram of the structure of the multi-scale feature fusion block provided by the present invention.

[0078] Figure 9 This is a structural diagram of a multispectral image fusion system based on spatial spectral difference prior guidance provided by the present invention.

[0079] Figure 10 This is a comparison chart of the fusion results of the present invention on the IKONOS dataset.

[0080] Reference numerals:

[0081] 101. Acquisition module; 102. Spatial difference feature construction module; 103. Spatial feature extraction module; 104. Spectral difference feature construction module; 105. Spectral feature extraction module; 106. Fusion module. DETAILED DESCRIPTION

[0082] To make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below. Obviously, the embodiments described are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0083] In the description of the embodiments of the present invention, it should be noted that the terms "first", "second" and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. The description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0084] The following combination Figures 1 to 10 The present invention describes a multispectral image fusion method and system based on spatial spectral difference prior guidance.

[0085] like Figure 1 As shown in FIG, a multispectral image fusion method based on spatial spectral difference prior guidance includes:

[0086] S1: Acquire LRMS images and panchromatic images through remote sensing optical imaging system;

[0087] Acquire images of remote sensing optical imaging systems in the fields of mineral detection, ecosystem monitoring, agricultural detection, etc., and obtain LRMS images and full-color images.

[0088] Model framework such as Figure 2 As shown, S2: upsample the LRMS image to obtain an upsampled LRMS image of the same size as the panchromatic image, subtract the upsampled LRMS image from the panchromatic image band by band to obtain a spatial difference feature map, and splice the spatial difference feature map with the panchromatic image to obtain a spatial difference input feature map;

[0089] The calculation expression is:

[0090]

[0091] in, Input feature map for spatial differences, For channel splicing operation, is a full-color image, is the first upsampled LRMS image bands, is the number of bands of the upsampled LRMS image.

[0092] S3: Perform multi-scale feature extraction on the spatial difference input feature map through the spatial difference prior guided branch (Spa-prior branch) to obtain a multi-scale spatial feature map;

[0093] The spatial difference prior-guided branch includes spatial feature extraction layers at multiple scales. The first-scale spatial feature extraction layer includes a 3×3 convolutional layer and a spatial feature extraction block (SpaFEB), and the remaining scale spatial feature extraction layers include downsampling, spatial feature extraction blocks, and prior-guided feature enhancement blocks (PFEB).

[0094] In some specific embodiments of the present invention, the spatial difference prior guidance branch uses the spatial difference feature as a priori to guide the model to extract spatial detail features at different scales. The feature maps are sequentially fed into the feature extraction layers at three scales. The first scale consists of a 3×3 convolutional layer and a spatial feature extraction module. The feature maps are processed by downsampling by a factor of two using bicubic interpolation before entering the spatial feature extraction modules at the second and third scales. A priori-guided feature enhancement block is introduced after the spatial feature extraction module to further exploit the difference information.

[0095] The spatial difference prior guide branch is used to extract multi-scale features from the spatial difference input feature map, and the calculation expression for the multi-scale spatial feature map is:

[0096]

[0097]

[0098] in, is the first scale space feature map, is the spatial feature extraction block, is a 3×3 convolutional layer, Input feature map for spatial differences, is the full-color image of the first scale, For the Full-scale color image, For the Scale-space feature map, is a priori guided feature enhancement block, is a double downsampling operation, For the Scale-space feature map.

[0099] and All are obtained by downsampling by a factor of two.

[0100] like Figure 3 As shown in the figure, the spatial feature extraction block consists of two convolutional layers, two activation layers, and a spatial feature enhancement unit (SpaEU). It aims to enhance spatial features at various scales by leveraging the features of the full-color image. In the Spatial Feature Enhancement Unit, the high-frequency features of the full-color image and the spatial features at various scales are used to generate a spatial weight map to enhance the spatial features.

[0101] Specifically, no. Full-color image at scale First, high-pass filter is used to extract high-frequency information, and the output features of two convolutional layers and two activation layers are obtained. Features are extracted through global maximum pooling (GMP) in the channel dimension. Features are extracted through global average pooling (GAP) in the channel dimension, and these three features are concatenated and sent to a 7×7 convolution layer and GELU activation function to obtain a spatial weight map , The dimension is , and After element-wise multiplication, Add to obtain enhanced spatial features, The dimension is , is the height, is the width;

[0102] The calculation expression is:

[0103]

[0104]

[0105]

[0106] in, is the output feature after two convolutional layers and two activation layers, is the GELU activation function, is a 7×7 convolution, For channel splicing operation, The input features of the spatial feature extraction block, is the spatial weight map, For the Full-scale color image, is a high-pass filter, is the global average pooling, is the global maximum pooling operation, For the The output features of the scale-space feature extraction block, is element-wise multiplication, The output features of the first scale-space feature extraction block.

[0107] In some specific embodiments of the present invention, the input of the spatial feature extraction block on the first scale is the original input feature of the spatial difference prior guided branch obtained by 3×3 convolution processing, and the input of the spatial feature extraction block on the second scale is After two times downsampling, the input of the spatial feature extraction block on the third scale is the second scale spatial feature map After two times downsampling, at the first scale, a 3×3 convolutional layer is used before the spatial feature extraction block, and no prior guided feature enhancement block is used, so .

[0108] like Figure 4 As shown in Figure 2, the prior-guided feature enhancement block uses the spatial difference prior information and the full-color image information to further enhance and correct the spatial features. The prior-guided feature enhancement block includes:

[0109] The spatial difference input feature map is downsampled and passed through 3 3 convolutional layers to obtain supplementary features;

[0110] The first The output features of the scale space feature extraction block are subtracted from the supplementary features to obtain the residual features;

[0111] The residual features are passed through 3 3. Convolutional layer and Sigmoid function processing to obtain the difference weight map;

[0112] The difference weight map and Multiply the output features of the scale space feature extraction block to obtain detail enhancement features;

[0113] The detail enhancement feature is concatenated with the supplementary feature, and then fused with the detail enhancement feature after channel attention and convolution operations to obtain the first Scale-space feature map;

[0114] The computational expression of the prior-guided feature enhancement block is:

[0115]

[0116]

[0117]

[0118]

[0119] in, To supplement the features, for Convolutional layers, is the downsampling operation, is the difference weight graph, is the Sigmoid activation function, Enhance features for detail, for Convolutional layers, It is the channel attention mechanism.

[0120] S4: downsample and upsample the panchromatic image to obtain a degraded panchromatic image, subtract the upsampled LRMS image from the degraded panchromatic image band by band to obtain a spectral difference feature map, and concatenate the spectral difference feature map with the upsampled LRMS image to obtain a spectral difference input feature map;

[0121]

[0122] in, Input feature map for spectral differences, is four times downsampling, is four times upsampling, is the upsampled LRMS image, A full-color image.

[0123] S5: Perform multi-scale feature extraction on the spectral difference input feature map through the spectral difference prior guided branch (Spe-prior branch) to obtain a multi-scale spectral feature map;

[0124] The spectral difference prior guidance branch uses spectral difference features as prior guidance to learn spectral features at different scales.

[0125] The spectral difference prior-guided branch includes spectral feature extraction layers at multiple scales. The first-scale spectral feature extraction layer includes a 3×3 convolutional layer and a spectral feature extraction block (SpeFEB). The remaining scale spectral feature extraction layers include downsampling, spectral feature extraction blocks, and prior-guided feature enhancement blocks.

[0126] The calculation expression is:

[0127]

[0128]

[0129] in, is the first scale spectral feature map, is the spectral feature extraction block, is a 3×3 convolutional layer, Input feature map for spectral difference, is the upsampled LRMS image, For the Scale spectrum characteristic diagram, is a priori guided feature enhancement block, is a double downsampling operation, For the Scale spectral feature map.

[0130] The spectral feature extraction block utilizes the features of low-resolution multispectral images to supplement and enhance the spectral features at various scales.

[0131] like Figure 5 As shown in the figure, the spectral feature extraction block includes a spectral feature enhancement unit and two spectral feature extraction units (SpeEU), which are composed of Convolutional layer, The convolution layer and GELU activation function are used to extract multi-scale features and fuse them. In the spectral feature enhancement unit, by combining low-resolution multispectral images and the spectral features at each scale, and calculate the spectral weight vector to enhance the spectral information.

[0132] Specifically, the low-resolution multispectral image and the features output by the two spectral feature extraction units are Perform global average pooling (GAP) and multi-layer perceptron (MLP) operations to obtain the spectral weight vector, the spectral weight vector and After channel-by-channel multiplication, Add together to get the enhanced spectral features, and the dimension of the spectral weight vector is , is the number of channels.

[0133] The calculation expression of the spectral feature extraction block is:

[0134]

[0135]

[0136]

[0137] in, Output features for the spectral feature extraction unit, is the spectral feature extraction unit, is the input feature of the spectral feature extraction block, is a 7×7 convolution, is the GELU activation function, is a multi-layer perceptron, is the global average pooling operation, Element-wise multiplication, For the The output features of the scale spectrum feature extraction block, The output features of the first-scale spectral feature extraction block.

[0138] In some specific embodiments of the present invention, the input of the spectral feature extraction block at the first scale is the original input feature of the spectral difference prior guided branch obtained by 3×3 convolution processing, the input of the second scale spectral feature extraction layer is the first scale spectral feature map obtained by double downsampling, and the input of the third scale spectral feature extraction layer is the second scale spectral feature map obtained by double downsampling. At the first scale, a 3×3 convolution layer is used before the spectral feature extraction layer, and the prior guided feature enhancement block is not used, so .

[0139] In the spectral difference prior guidance branch, the prior guidance feature enhancement block includes:

[0140] The spectral difference is input into the feature map for downsampling and is passed through 3 3 convolutional layers to obtain supplementary features;

[0141] The first The output features of the scale spectrum feature extraction block are subtracted from the supplementary features to obtain the residual features;

[0142] The residual features are passed through 3 3. Convolutional layer and Sigmoid function processing to obtain the difference weight map;

[0143] The difference weight map and Multiply the output features of the scale spectrum feature extraction block to obtain detail enhancement features;

[0144] The detail enhancement feature is concatenated with the supplementary feature, and then fused with the detail enhancement feature after channel attention and convolution operations to obtain the first Scale spectral feature map.

[0145] S6: The multi-scale spatial feature map and the multi-scale spectral feature map are fused through the spatial-spectral feature fusion branch to obtain the HRMS image.

[0146] In order to achieve the fusion of spatial and spectral features from the spatial difference prior guidance branch and the spectral difference prior guidance branch at different scales, a spatial-spectral feature fusion branch (Spa-Spe-fusion branch) was constructed.

[0147] Firstly, a spatial-spectral feature fusion block (SFFB) is designed at each scale layer to fuse spatial and spectral features of the same scale. At the same time, a cross-scale feature fusion block (CFFB) is constructed to integrate the fusion features of adjacent scales. The multi-scale fusion features are then input into the multi-scale feature fusion block (MFFB) to obtain the HRMS image.

[0148] The calculation expression is:

[0149]

[0150]

[0151]

[0152]

[0153] in, For the Spatial feature maps of scale, For the Spectral characteristic diagram of scale, is the spatial-spectral feature fusion block, is the cross-scale feature fusion block, It is a multi-scale feature fusion block operation. The spatial-spectral feature fusion block is Output on scale, For the cross-scale feature fusion block in Output on scale, High-resolution multispectral images.

[0154] like Figure 6 As shown, the spatial-spectral feature fusion block fuses the spatial feature map and the spectral feature map of the same scale;

[0155] The first The spatial feature map of the scale and the The scale spectral feature maps are spliced ​​to obtain the scale-space spectral feature maps;

[0156] No. The scale space spectral feature map is generated by 3×3 convolution layer, GELU activation function, 3×3 convolution layer and Sigmoid function. Scale-space feature weight map;

[0157] Through the The scale space feature weight map is The spatial feature map of the scale is multiplied element by element and then Add the scale spectral feature maps to obtain the Scale-space spectral fusion feature map;

[0158] The calculation expression is:

[0159]

[0160]

[0161] in, is the spatial feature weight map, The spatial-spectral feature fusion block is Output on scale.

[0162] like Figure 7 As shown in Figure 2, the cross-scale feature fusion block achieves the fusion of adjacent scale features by establishing a global correlation between cross-scale features.

[0163] The first The scale space spectrum fusion feature is upsampled to the Scale-space spectral fusion features the same size,

[0164] The spatial-spectral fusion features of the two scales are reshaped into tokens and normalized before being fed into the linear cross attention unit and combined with the first Add up the scale tokens to get the Scale Enhancement Token,

[0165] No. After the scale-enhanced tokens are reshaped into feature maps, The scale-space spectral fusion features are added to obtain the cross-scale feature fusion block in the first i Output on scale;

[0166] The calculation expression is:

[0167]

[0168]

[0169]

[0170]

[0171] in, For the scale token, For the reshape operation, For the scale token, The spatial-spectral feature fusion block is Output on scale, is the layer normalization (LayerNorm) operation, is the operation of the linear cross attention unit, For the Token after scale enhancement, For the cross-scale feature fusion block in Output on scale.

[0172] The linear crisscross attention unit is:

[0173] The first The token map after scale layer normalization is the query matrix,

[0174] No. The scale-normalized tokens enter two branches,

[0175] The first branch passes through the linear layer, The convolution and SiLU activation functions aggregate local information to obtain a value matrix, which is further mapped to a key matrix through a linear layer. The linear attention is calculated using the query matrix, value matrix, and key matrix to obtain the linear attention output.

[0176] The second branch passes through the linear layer and SiLU activation mapping, multiplies it with the linear attention output, and passes through a linear layer mapping to obtain the output of the linear cross attention unit;

[0177] The calculation expression is:

[0178]

[0179] in, is the query matrix, is a linear layer, For the Tokens after scale layer normalization, is the value matrix, is the SiLU activation function, For the Tokens after scale layer normalization, is the bond matrix, is the transpose of the matrix, represents matrix multiplication, is the output of the linear crisscross attention unit.

[0180] like Figure 8 As shown in Figure 1, the multi-scale feature fusion block is used to generate a high-resolution multispectral image. The fused features of the third scale are upsampled and integrated with the features of the second scale through addition operations. The integrated features are further processed by a 3×3 convolution layer. Subsequently, the integrated features are upsampled again and added to the features of the first scale to obtain a rough fused feature. Finally, the 3×3 convolution layer and Convolution is performed to obtain the final fusion result.

[0181] The calculation expression is:

[0182]

[0183] in, For high-resolution multispectral images, is a 3×3 convolutional layer, is a 1×1 convolutional layer, is a double upsampling operation, For the cross-scale feature fusion block in Output on scale, .

[0184] like Figure 9 As shown, a multispectral image fusion system based on spatial spectral difference prior guidance is used to perform a multispectral image fusion method based on spatial spectral difference prior guidance, including:

[0185] The acquisition module 101 acquires the LRMS image and the panchromatic image through the remote sensing optical imaging system;

[0186] The spatial difference feature construction module 102 upsamples the LRMS image to obtain an upsampled LRMS image of the same size as the panchromatic image, subtracts the upsampled LRMS image from the panchromatic image band by band to obtain a spatial difference feature map, and concatenates the spatial difference feature map with the panchromatic image to obtain a spatial difference input feature map.

[0187] The spatial feature extraction module 103 performs multi-scale feature extraction on the spatial difference input feature map through the spatial difference prior guidance branch to obtain a multi-scale spatial feature map;

[0188] The spectral difference feature construction module 104 downsamples and upsamples the panchromatic image to obtain a degraded panchromatic image, subtracts the upsampled LRMS image from the degraded panchromatic image band by band to obtain a spectral difference feature map, and concatenates the spectral difference feature map with the upsampled LRMS image to obtain a spectral difference input feature map;

[0189] The spectral feature extraction module 105 performs multi-scale feature extraction on the spectral difference input feature map through the spectral difference prior guidance branch to obtain a multi-scale spectral feature map;

[0190] The fusion module 106 fuses the multi-scale spatial feature map and the multi-scale spectral feature map through the spatial-spectral feature fusion branch to obtain the HRMS image.

[0191] Through the collaborative work of the above modules, the spectral and spatial difference information of the panchromatic image and the LRMS image at the pixel level is used as a priori to guide the extraction and fusion of complementary information, which significantly improves the accuracy of feature extraction and the fusion effect, and realizes accurate reconstruction of high-resolution multispectral images.

[0192] To verify the superiority of the present invention, quantitative and subjective comparative experiments were conducted with a variety of internationally advanced methods, including PSCF-Net (Deep Coupled Feedback Network for Pan-Sharpening, TGRS 2023 Edition), BiMPan (Dual Domain Modeling Framework for Pan-Sharpening, ACM MM 2023 Edition), DISPNet (Intrinsically Supervised Deep Unfolding Network for Pan-Sharpening, AAAI 2024 Edition), and WFANet (Wavelet-Assisted Multi-Frequency Attention Network for Pan-Sharpening, AAAI 2025 Edition). As shown in Table 1, the quantitative indicator comparison results on the WorldView-3 dataset show that the present invention achieves the best results in all indicators.

[0193] Table 1 Comparison of quantitative indicators on the WorldView-3 dataset

[0194]

[0195] In the table, SCC is the spatial correlation coefficient, SSIM is the structural similarity index, SAM is the spectral angle, UIQI is the universal image quality index, ERGAS is the comprehensive relative global dimensionless error and PSNR peak signal-to-noise ratio, "↑" means the higher the value, the better, "↓" means the lower the value, the better, and the best indicators are marked in bold black.

[0196] like Figure 10 The figure shows the fusion results on the IKONOS dataset. The red box represents a magnified area. To more intuitively demonstrate the difference between the fusion result and the ground truth (GT), a residual image was calculated and visualized, and the corresponding residual image for the red boxed area is displayed in the yellow box. The results show that the fusion result of the present invention is closest to the ground truth, and the residual image has more dark blue areas, indicating a smaller fusion error. Qualitative and quantitative analysis demonstrate that the comprehensive performance of the present invention outperforms recent internationally advanced methods.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multispectral image fusion method based on spatial spectral difference prior guidance, characterized in that: include: S1: Acquire LRMS images and panchromatic images through remote sensing optical imaging system; S2: Upsample the LRMS image to obtain an upsampled LRMS image of the same size as the panchromatic image, subtract the upsampled LRMS image from the panchromatic image band by band to obtain a spatial difference feature map, and concatenate the spatial difference feature map with the panchromatic image to obtain a spatial difference input feature map; S3: Multi-scale feature extraction is performed on the spatial difference input feature map through the spatial difference prior guidance branch to obtain a multi-scale spatial feature map; The spatial difference prior guided branch includes spatial feature extraction layers at multiple scales. The first scale spatial feature extraction layer includes a 3×3 convolutional layer and a spatial feature extraction block. The remaining scale spatial feature extraction layers include downsampling, spatial feature extraction blocks, and prior guided feature enhancement blocks. The spatial difference prior guide branch is used to extract multi-scale features from the spatial difference input feature map, and the calculation expression for the multi-scale spatial feature map is: in, is the first scale space feature map, is the spatial feature extraction block, is a 3×3 convolutional layer, Input feature map for spatial differences, is the full-color image of the first scale, For the Full-scale color image, For the Scale-space feature map, is a priori guided feature enhancement block, is a double downsampling operation, For the Scale-space feature map; S4: downsample and upsample the panchromatic image to obtain a degraded panchromatic image, subtract the upsampled LRMS image from the degraded panchromatic image band by band to obtain a spectral difference feature map, and concatenate the spectral difference feature map with the upsampled LRMS image to obtain a spectral difference input feature map; S5: extract multi-scale features from the spectral difference input feature map through the spectral difference prior guided branch to obtain a multi-scale spectral feature map; The spectral difference prior guidance branch includes spectral feature extraction layers of multiple scales, the first scale spectral feature extraction layer includes a 3×3 convolution layer and a spectral feature extraction block, and the remaining scale spectral feature extraction layers include downsampling, spectral feature extraction blocks and prior guidance feature enhancement blocks; The spectral difference prior guide branch is used to extract multi-scale features from the spectral difference input feature map, and the calculation expression for the multi-scale spectral feature map is: in, is the first scale spectral feature map, is the spectral feature extraction block, Input feature map for spectral difference, is the upsampled LRMS image, For the Scale spectral feature map, For the scale spectral feature map; The prior-guided feature enhancement block includes: The difference input feature map is downsampled and passed through 3 3 convolutional layers to obtain supplementary features; The first The output features of the scale feature extraction block are subtracted from the supplementary features to obtain the residual features; The residual features are passed through 3 3. Convolutional layer and Sigmoid function processing to obtain the difference weight map; The difference weight map and Multiply the output features of the scale feature extraction block to obtain detail enhancement features; The detail enhancement features are concatenated with the supplementary features and fused with the detail enhancement features through channel attention and convolution operations to obtain the first scale feature map; Among them, in step S3, the difference input feature map is the spatial difference input feature map, The scale feature extraction block is Scale space feature extraction block, The scale feature map is Scale-space feature map; In step S5, the difference input feature map is the spectral difference input feature map. The scale feature extraction block is Scale spectrum feature extraction block, The scale feature map is scale spectral feature map; S6: The multi-scale spatial feature map and the multi-scale spectral feature map are fused through the spatial-spectral feature fusion branch to obtain the HRMS image.

2. The multispectral image fusion method based on spatial spectrum difference prior guidance according to claim 1 is characterized in that: The spatial feature extraction block includes two convolutional layers, two activation layers, and a spatial feature enhancement unit. The calculation expression is: in, is the output feature after two convolutional layers and two activation layers, is the GELU activation function, is a 7×7 convolution, For channel splicing operation, The input features of the spatial feature extraction block, is the spatial weight map, For the Full-scale color image, is a high-pass filter, is the global average pooling, is the global maximum pooling operation, For the The output features of the scale-space feature extraction block, is element-wise multiplication, The output features of the first scale-space feature extraction block.

3. The multispectral image fusion method based on spatial spectrum difference prior guidance according to claim 1 is characterized in that: The spectral feature extraction block includes a spectral feature enhancement unit and two spectral feature extraction units. The calculation expression is: in, Output features for the spectral feature extraction unit, is the spectral feature extraction unit, is the input feature of the spectral feature extraction block, is a 7×7 convolution, is the GELU activation function, is a multi-layer perceptron, is the global average pooling operation, Element-wise multiplication, For the The output features of the scale spectrum feature extraction block, The output features of the first-scale spectral feature extraction block.

4. The multispectral image fusion method based on spatial spectrum difference prior guidance according to claim 1 is characterized in that: The spatial-spectral feature fusion branch includes a spatial-spectral feature fusion block, which fuses the spatial feature map and the spectral feature map of the same scale; The first Scale space feature map and The scale spectral feature maps are spliced ​​to obtain the scale-space spectral feature maps; No. The scale space spectral feature map is generated by 3×3 convolution layer, GELU activation function, 3×3 convolution layer and Sigmoid function. Scale-space feature weight map; Through the Scale space feature weight map and the The spatial feature map of the scale is multiplied element by element and then Add the scale spectral feature maps to obtain the Scale-space spectral fusion feature map.

5. The multispectral image fusion method based on spatial spectrum difference prior guidance according to claim 1, characterized in that: The spatial-spectral feature fusion branch also includes a cross-scale feature fusion block, which achieves the fusion of adjacent scale features by establishing a global correlation between cross-scale features. The first The scale space spectrum fusion feature is upsampled to the Scale space spectral fusion features the same size; The spatial-spectral fusion features of the two scales are reshaped into tokens and normalized before being fed into the linear cross attention unit and combined with the first Add up the scale tokens to get the Scale Enhancement Token; No. After the scale-enhanced tokens are reshaped into feature maps, The scale-space spectral fusion features are added to obtain the cross-scale feature fusion block in the first i Output on scale; Among them, the linear cross attention unit is: The first The scale-normalized token mapping is the query matrix; No. The scale-normalized tokens enter two branches, The first branch aggregates local information through a linear layer, a 3×3 convolution, and a SiLU activation function to obtain a value matrix, which is further mapped to a key matrix through a linear layer. The linear attention is calculated using the query matrix, the value matrix, and the key matrix to obtain the linear attention output; The second branch passes through the linear layer and SiLU activation mapping, multiplies it with the linear attention output, and passes through a linear layer mapping to obtain the output of the linear cross attention unit.

6. The multispectral image fusion method based on spatial spectrum difference prior guidance according to claim 1, characterized in that: The spatial-spectral feature fusion branch also includes a multi-scale feature fusion block, which is used to generate a high-resolution multispectral image. The calculation expression is: in, For high-resolution multispectral images, is a 3×3 convolutional layer, is a 1×1 convolutional layer, is a double upsampling operation, For the cross-scale feature fusion block in Output on scale, .

7. A multispectral image fusion system based on spatial spectral difference prior guidance, characterized in that: The method for performing a multispectral image fusion method based on spatial spectral difference prior guidance according to any one of claims 1 to 6 comprises: An acquisition module, wherein the acquisition module acquires an LRMS image and a panchromatic image through a remote sensing optical imaging system; A spatial difference feature construction module is configured to upsample the LRMS image to obtain an upsampled LRMS image of the same size as the panchromatic image; subtract the upsampled LRMS image from the panchromatic image band by band to obtain a spatial difference feature map; and concatenate the spatial difference feature map with the panchromatic image to obtain a spatial difference input feature map. A spatial feature extraction module, wherein the spatial feature extraction module performs multi-scale feature extraction on the spatial difference input feature map through a spatial difference prior guided branch to obtain a multi-scale spatial feature map; a spectral difference feature construction module, which downsamples and upsamples the panchromatic image to obtain a degraded panchromatic image, subtracts the upsampled LRMS image from the degraded panchromatic image band by band to obtain a spectral difference feature map, and concatenates the spectral difference feature map with the upsampled LRMS image to obtain a spectral difference input feature map; A spectral feature extraction module, wherein the spectral feature extraction module performs multi-scale feature extraction on the spectral difference input feature map through a spectral difference prior guided branch to obtain a multi-scale spectral feature map; The fusion module fuses the multi-scale spatial feature map and the multi-scale spectral feature map through a spatial-spectral feature fusion branch to obtain an HRMS image.

Images