A method and device for multi-scale fusion of hyperspectral images and multispectral images

Abstract

The present invention discloses a method and device for multi-scale fusion of hyperspectral images and multispectral images in the field of hyperspectral image processing technology, aiming to solve the problems in the prior art of insufficient preservation of spectral and spatial information and poor spatial resolution of the fusion results when processing hyperspectral and multispectral images. The method obtains a hyperspectral image and a multispectral image; splices and fuses the upsampled hyperspectral image with the multispectral image to obtain a fused hyperspectral image; extracts features from the fused hyperspectral image to obtain shallow spectral spatial features; and performs wavelet transform on the shallow spectral spatial features to obtain a multi-scale sub-band feature map. The present invention realizes multi-scale feature extraction through wavelet transform and adds a local feature extraction module to the Mamba module to fully obtain spectral and spatial information, thereby realizing multi-scale fusion of hyperspectral and multispectral images, improving the accuracy of feature extraction, and obtaining higher quality images.

Description

Technical Field

[0001] The present invention relates to a method and device for multi-scale fusion of hyperspectral images and multispectral images, belonging to the technical field of hyperspectral image processing. Background Art

[0002] Hyperspectral imagery (HSI) and multispectral imagery (MSI) have widespread applications in remote sensing, environmental monitoring, agriculture, geological exploration, and other fields. Hyperspectral imagery provides rich spectral information, but its spatial resolution is relatively low. In contrast, multispectral imagery has higher spatial resolution but relatively less spectral information. Therefore, how to effectively fuse hyperspectral and multispectral images to obtain images with both high spatial and spectral resolution has become an important research topic.

[0003] Traditional image fusion methods are mainly divided into two categories: spatial domain and transform domain. Spatial domain methods achieve fusion by directly manipulating pixel values, but they struggle to fully preserve spectral and spatial information. Transform domain methods fuse by converting images into other domains. While they can better preserve details, they are computationally complex when processing high-dimensional data and struggle to cope with nonlinear relationships. In recent years, deep learning-based image fusion methods have made significant progress, but they rely on large amounts of training data, have high model complexity, and are difficult to adapt to small sample scenarios. In addition, existing methods often ignore multi-scale characteristics when processing hyperspectral and multispectral images, resulting in insufficient preservation of spectral and spatial information in the fusion results and poor spatial resolution. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method and device for multi-scale fusion of hyperspectral images and multispectral images. Multi-scale feature extraction is achieved through wavelet transform, and a local feature extraction module is added to the Mamba module to fully obtain spectral information and spatial information. Multi-scale fusion of hyperspectral images and multispectral images is achieved, the accuracy of feature extraction is improved, and higher-quality high-resolution hyperspectral images are obtained. The problem of low fusion accuracy caused by insufficient retention of spectral information and spatial information in the process of fusing multi-source remote sensing images is solved.

[0005] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a method for multi-scale fusion of hyperspectral images and multispectral images, comprising the following steps:

[0007] Acquire hyperspectral and multispectral images;

[0008] Upsampling the hyperspectral image to obtain an upsampled hyperspectral image, and concatenating and fusing the upsampled hyperspectral image with the multispectral image to obtain a fused hyperspectral image;

[0009] Perform feature extraction on the fused hyperspectral image to obtain shallow spectral spatial features;

[0010] Perform wavelet transform on shallow spectral spatial features to obtain multi-scale sub-band feature maps;

[0011] The multi-scale sub-band feature map is input into the deep feature extraction module to obtain the multi-scale deep spectral spatial features;

[0012] Fusing multi-scale deep spectral spatial features to obtain fused deep spectral spatial features, and obtaining spectral spatial features based on the fused deep spectral spatial features and shallow spectral spatial features;

[0013] According to the spectral spatial characteristics and the upsampled hyperspectral image, a high-resolution hyperspectral image is obtained.

[0014] Furthermore, upsampling the hyperspectral image to obtain the upsampled hyperspectral image specifically includes:

[0015] Perform bilinear interpolation upsampling on the hyperspectral image to obtain an upsampled hyperspectral image;

[0016] The step of splicing and fusing the upsampled hyperspectral image with the multispectral image to obtain a fused hyperspectral image specifically includes:

[0017] The multispectral image is stitched and fused with the upsampled hyperspectral image using a stitching and fusion operation to obtain a fused hyperspectral image.

[0018] Furthermore, the fused hyperspectral image is subjected to feature extraction to obtain shallow spectral spatial features, which are expressed as follows:

[0019] ;

[0020] in, It is the shallow spectral spatial feature; is the fused hyperspectral image; It is a shallow feature extraction module, which includes convolution and multiple residual blocks. The convolution is used to perform preliminary feature extraction on the input initial features to obtain the first extracted features; the multiple residual blocks are used to extract higher-level features layer by layer from the first extracted features to obtain the second extracted features, and then the initial features are residually connected with the second extracted features to obtain shallow spectral spatial features.

[0021] Furthermore, the shallow spectral spatial features are subjected to wavelet transformation to obtain a multi-scale sub-band feature map, specifically including:

[0022] Select Haar wavelet as the mother wavelet;

[0023] The input shallow spectral spatial features are low-pass and high-pass filtered in the horizontal and vertical directions respectively to generate a multi-scale sub-band feature map. The multi-scale sub-band feature map includes Subband feature map, Subband feature map, Subband feature map and Subband feature map, where The sub-band feature map represents the low-frequency component; The sub-band feature map represents the high-frequency component in the horizontal direction; The sub-band feature map represents the high-frequency component in the vertical direction; The sub-band feature map represents the high-frequency components in the diagonal direction.

[0024] Furthermore, the deep feature extraction module includes a spectral Mamba block and a spatial Mamba block;

[0025] The multi-scale sub-band feature map is input into the deep feature extraction module to obtain the multi-scale deep spectral spatial feature, which specifically includes performing the following steps on the sub-band feature map of each scale:

[0026] Step A: Flatten the subband feature map into a vector form and serialize the vector to obtain a sequence;

[0027] Step B: Generate position codes through the encoder and add the position codes to the sequence, retaining the spatial information of the sequence and obtaining the sequence with the correct encoded position information;

[0028] Step C: Input the sequence with correctly encoded position information into the spectral Mamba block and obtain deep spectral features through the decoder;

[0029] The subband feature map in step A is replaced by the deep spectral feature, and then steps A and B are repeated to re-obtain the sequence after the correct encoding position information is obtained. The sequence after the correct encoding position information is input into the spatial Mamba block, and the deep spectral spatial feature is obtained through the decoder.

[0030] Furthermore, the sequence after correctly encoding the position information is input into the spectral Mamba block, and the deep spectral features are obtained through the decoder, specifically including:

[0031] Step a: Sequence after correctly encoding position information Perform linear projection, and the projected features are divided into two feature parts along the channel dimension and , q is processed by the activation function, p is processed by the convolution to extract spatial features and then the activation function is processed to achieve feature block processing and spatial feature enhancement:

[0032] ;

[0033] ;

[0034] ;

[0035] in, For the projection operation, is the activation function, To correctly encode the sequence of position information, is the convolution operation, R is the scale, B is the batch, H is the height of the multispectral image, W is the width of the multispectral image, and C is the number of channels of the convolution in the shallow feature extraction module;

[0036] Step b: The feature part p generates a spectrum token M Or generate space token N , focusing on spectral features, the spectral token captures global information through Mamba's state space model and obtains local information through the window scanning mechanism:

[0037] ;

[0038] ;

[0039] in, To map all the information of each element from different directions from the four corners of the feature map to the relative position, It is a window scanning mechanism that captures local information between sequence elements. Represents the global information gathered by the four-way scanning strategy, Represents the local information gathered by window scanning, and The two are input together into the selective scanning spatial state sequence model and processed in parallel to obtain the intermediate deep spectral feature out_y;

[0040] Step c: Normalize the intermediate deep spectral features and compare Multiply the elements of each part to perform feature weighting, highlight the important features needed, and obtain the intermediate features:

[0041] ;

[0042] in, is the intermediate feature, is a linear layer operation, is the layer normalization operation;

[0043] Step d: Based on the intermediate features, the decoder is used to obtain the deep spectral features. ;

[0044] The sequence of correctly encoded position information is input into the spatial Mamba block, and deep spectral spatial features are obtained through the decoder, specifically including:

[0045] According to the deep spectral characteristics , re-obtain the sequence after correctly encoding the position information;

[0046] Replace the sequence of correctly encoded position information in step a with the sequence of correctly encoded position information that has been re-obtained, and then repeat steps a, b, and c to re-obtain the intermediate features;

[0047] Based on the re-acquired intermediate features, deep spectral spatial features are obtained through the decoder.

[0048] Furthermore, the fusion of multi-scale deep spectral spatial features specifically includes:

[0049] The multi-scale deep spectral spatial features are reconstructed into the original signal through the inverse two-dimensional discrete wavelet transform to obtain the fused deep spectral spatial features. The expression is as follows:

[0050] ;

[0051] in, is the deep spectral spatial feature after fusion; is the inverse two-dimensional discrete wavelet transform; Represents the deep spectral spatial features obtained by the cA subband feature map, represents the deep spectral spatial features obtained by the cH subband feature map, Represents the deep spectral spatial features obtained by the cV sub-band feature map, Represents the deep spectral spatial features obtained by the cD sub-band feature map.

[0052] Furthermore, the obtaining of spectral spatial features based on the fused deep spectral spatial features and shallow spectral spatial features specifically includes:

[0053] The spectral spatial features are obtained by using jump connections to aggregate the shallow spectral spatial features and the fused deep spectral spatial features.

[0054] Furthermore, obtaining a high-resolution hyperspectral image based on the spectral spatial characteristics and the upsampled hyperspectral image specifically includes:

[0055] Input the spectral spatial features into the image reconstruction module to obtain a high-resolution reconstructed image;

[0056] The high-resolution reconstructed image and the upsampled hyperspectral image are connected using residuals to fuse low-resolution and high-resolution information to obtain a high-resolution hyperspectral image.

[0057] In a second aspect, the present invention provides a device for multi-scale fusion of hyperspectral images and multispectral images, the device comprising:

[0058] Preliminary fusion module: obtain hyperspectral images and multispectral images; upsample the hyperspectral images to obtain upsampled hyperspectral images; stitch and fuse the upsampled hyperspectral images with the multispectral images to obtain fused hyperspectral images;

[0059] Shallow feature extraction module: extracts features from the fused hyperspectral image to obtain shallow spectral spatial features;

[0060] Deep feature extraction module: performs wavelet transform on shallow spectral spatial features to obtain multi-scale sub-band feature maps; inputs the multi-scale sub-band feature maps into the deep feature extraction module to obtain multi-scale deep spectral spatial features;

[0061] Fusion and reconstruction module: Fusion of multi-scale deep spectral spatial features to obtain fused deep spectral spatial features, and spectral spatial features based on the fused deep spectral spatial features and shallow spectral spatial features; and high-resolution hyperspectral images based on the spectral spatial features and the upsampled hyperspectral images.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] The present invention proposes a multi-scale fusion method and device based on the Mamba model and wavelet transform. The deep feature extraction module can efficiently process long sequence data. The wavelet transform can effectively capture the local features of the image through multi-scale analysis, thereby realizing the multi-scale fusion of hyperspectral images and multi-spectral images. While improving the spatial resolution, the spectral information is fully retained. The method proposed in the present invention realizes multi-scale feature extraction through wavelet transform, and fully obtains spectral information and spatial information according to the deep feature extraction module, thereby realizing the multi-scale fusion of hyperspectral images and multi-spectral images, improving the accuracy of feature extraction, obtaining higher-quality high-resolution hyperspectral images, and solving the problem of low fusion accuracy caused by insufficient retention of spectral information and spatial information in the process of fusing multi-source remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 1 is a flow chart of a method for multi-scale fusion of hyperspectral images and multispectral images according to an embodiment of the present invention;

[0065] Figure 2 1 is a schematic diagram of a framework of a method for multi-scale fusion of hyperspectral images and multispectral images provided by an embodiment of the present invention;

[0066] Figure 3 is a schematic diagram of a framework of a Mamba block provided according to an embodiment of the present invention;

[0067] Figure 4 RGB heat map of the hyperspectral image of the CAVE dataset provided in the embodiment of the present invention (with bands 10, 20, and 30 as RGB image data);

[0068] Figure 5 RGB heat map of the multispectral image of the CAVE dataset provided by the embodiment of the present invention;

[0069] Figure 6 This is a diagram of the CAVE dataset fusion results provided by an embodiment of the present invention;

[0070] Figure 7 A schematic diagram of the structure of a device for multi-scale fusion of hyperspectral images and multispectral images provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0072] The term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " generally indicates an "or" relationship between the related objects.

[0073] Example 1:

[0074] like Figure 1 FIG. 1 is a flow chart of a method for multi-scale fusion of hyperspectral images and multispectral images provided by an embodiment of the present invention, comprising the following steps:

[0075] Acquire hyperspectral and multispectral images;

[0076] Specifically, obtain the training set and validation set of multispectral images, including:

[0077] The CAVE dataset (Columbia Aerial Videography and Photography Experimental Dataset) is obtained, which contains 32 indoor hyperspectral images, each with a size of 512 × 512 pixels, covering the wavelength range of 400–700 nm in 31 frequency bands.

[0078] Generate high-resolution multispectral images (HR-MSI) using the camera's spectral response function;

[0079] Generate low-resolution hyperspectral image LR-HSI through Gaussian filtering and multiple downsampling;

[0080] Randomly select 20 pairs of images from the dataset as training sets, and the remaining 12 pairs of images as validation sets;

[0081] In this embodiment, during the training phase, a 64×64 patch is randomly extracted from each 512×512 image and input into the model. That is, in the training phase, this embodiment takes H=64, W=64, s=3, h=8, w=8, and S=31, where h is the height of the hyperspectral image, w is the width of the hyperspectral image, S is the channel of the hyperspectral image, H is the height of the multispectral image, W is the width of the multispectral image, and s is the channel of the multispectral image.

[0082] like Figure 2 , which is a schematic diagram of a framework of a method for multi-scale fusion of hyperspectral images and multispectral images provided by an embodiment of the present invention;

[0083] Among them, low frequency refers to the low-frequency component, horizontal high frequency refers to the high-frequency component in the horizontal direction, vertical high frequency refers to the high-frequency component in the vertical direction, and diagonal high frequency refers to the high-frequency component in the diagonal direction.

[0084] In one embodiment, a hyperspectral image is upsampled to obtain an upsampled hyperspectral image, and the upsampled hyperspectral image is spliced and fused with a multispectral image to obtain a fused hyperspectral image.

[0085] The upsampling of the hyperspectral image to obtain the upsampled hyperspectral image specifically includes:

[0086] Perform bilinear interpolation upsampling on the hyperspectral image to obtain an upsampled hyperspectral image;

[0087] The step of splicing and fusing the upsampled hyperspectral image with the multispectral image to obtain a fused hyperspectral image specifically includes:

[0088] The multispectral image is stitched and fused with the upsampled hyperspectral image using a stitching and fusion operation to obtain a fused hyperspectral image.

[0089] Specifically, the hyperspectral image is , the multispectral image is Perform bilinear interpolation upsampling on the hyperspectral image to obtain the upsampled hyperspectral image, and then splice and fuse the multispectral image with the upsampled hyperspectral image to obtain the fused hyperspectral image. , R represents the scale.

[0090] Specifically, for hyperspectral images Perform upsampling operation to upsample the hyperspectral image With multispectral imagery Perform channel splicing and fusion, including:

[0091] , ;

[0092] ;

[0093] in, is the upsampling operation, For splicing and fusion operations, is the fused hyperspectral image.

[0094] In one embodiment, feature extraction is performed on the fused hyperspectral image to obtain shallow spectral spatial features;

[0095] Specifically, the fused hyperspectral image is subjected to feature extraction to obtain shallow spectral spatial features. , whose expression is:

[0096] ;

[0097] in, It is the shallow spectral spatial feature; is the fused hyperspectral image; It is a shallow feature extraction module, which includes convolution and multiple residual blocks. The convolution is used to perform preliminary feature extraction on the input initial features to obtain the first extracted features; the multiple residual blocks are used to extract higher-level features layer by layer from the first extracted features to obtain the second extracted features, and then the initial features are residually connected with the second extracted features to obtain shallow spectral spatial features.

[0098] Specifically, the shallow features of the fused hyperspectral image are extracted, and the initial features are extracted by convolution:

[0099] ;

[0100] The residual block then gradually extracts higher-level features and retains more detailed information, including:

[0101] ;

[0102] ;

[0103] ;

[0104] in, is the convolution operation, is a convolution operation with a convolution kernel of 3×3. represents the continuous composition of functions, is the i-th residual block, i is 1, 2, 3; It is the shallow spectral spatial feature; is the number of residual blocks, The C in the above figure represents the number of channels of the convolution in the shallow feature extraction module. In this paper, we take C=180. =3.

[0105] In one embodiment, the shallow spectral spatial features are subjected to wavelet transformation to obtain a multi-scale sub-band feature map;

[0106] The wavelet transform of the shallow spectral spatial features to obtain the multi-scale sub-band feature map specifically includes:

[0107] Select Haar wavelet as the mother wavelet;

[0108] The input shallow spectral spatial features are low-pass and high-pass filtered in the horizontal and vertical directions respectively to generate a multi-scale sub-band feature map. The multi-scale sub-band feature map includes Subband feature map, Subband feature map, Subband feature map and Subband feature map, where The sub-band feature map represents the low-frequency component; The sub-band feature map represents the high-frequency component in the horizontal direction; The sub-band feature map represents the high-frequency component in the vertical direction; The sub-band feature map represents the high-frequency components in the diagonal direction.

[0109] Specifically, the extracted shallow spectral spatial features are subjected to wavelet transform to obtain sub-band feature maps at four scales to obtain more local detail information, including:

[0110] ;

[0111] ;

[0112] in, For two-dimensional discrete wavelet transform, cA is the low-frequency component, cH is the high-frequency component in the horizontal direction, cV is the high-frequency component in the vertical direction, and cD is the high-frequency component in the diagonal direction. The mother wavelet function of the two-dimensional discrete wavelet transform is the Haar wavelet.

[0113] In one embodiment, the multi-scale sub-band feature map is input into a deep feature extraction module to obtain multi-scale deep spectral spatial features;

[0114] The deep feature extraction module includes a spectral Mamba block and a spatial Mamba block;

[0115] The multi-scale sub-band feature map is input into the deep feature extraction module to obtain the multi-scale deep spectral spatial feature, which specifically includes performing the following steps on the sub-band feature map of each scale:

[0116] Step A: Flatten the subband feature map into a vector form and serialize the vector to obtain a sequence;

[0117] Step B: Generate position codes through the encoder and add the position codes to the sequence, retaining the spatial information of the sequence and obtaining the sequence with the correct encoded position information;

[0118] Step C: Input the sequence with correctly encoded position information into the Spectral Mamba (Mamba Network) block, and use the decoder to obtain deep spectral features. The Spectral Mamba module is composed of multiple Mamba blocks. The core feature of the Mamba block is to use the state space model to efficiently model long sequence data and capture global dependencies. Through forward and backward scanning mechanisms, the global context modeling capability of the sequence data is enhanced. The introduction of window scanning enhances the modeling capability of local features, allowing the Spectral Mamba module to efficiently extract global and local features. Finally, through the decoder, a feature map containing spectral feature information is obtained.

[0119] The subband feature map in step A is replaced by the deep spectral feature, and then steps A and B are repeated to re-obtain a sequence after the correct encoding position information is obtained. The sequence after the correct encoding position information is input into the spatial Mamba block, and the deep spectral spatial feature is obtained through the decoder; the deep spectral feature is expanded using the same method to obtain a spatial sequence, which is input into the spatial Mamba module to obtain the spectral spatial feature. The Mamba module has the same structure as the spectral Mamba module, except that the spectral Mamba module focuses on spectral information while the spatial Mamba module focuses on spatial information.

[0120] In one embodiment, the sequence after correctly encoding the position information is input into the spectral Mamba block, and the deep spectral features are obtained through the decoder, specifically including:

[0121] Step a: Sequence after correctly encoding position information Perform linear projection, and the projected features are divided into two feature parts along the channel dimension and , q is processed by the activation function, p is processed by the convolution to extract spatial features and then the activation function is processed to achieve feature block processing and spatial feature enhancement:

[0122] ;

[0123] ;

[0124] ;

[0125] in, For the projection operation, is the activation function, To correctly encode the sequence of position information, is the convolution operation, R is the scale, B is the batch, H is the height of the multispectral image, W is the width of the multispectral image, and C is the number of channels of the convolution in the shallow feature extraction module;

[0126] Step b: The feature part p generates a spectrum token M Or generate space token N ,in, ], Indicates the spatial direction pixels; N=[ ], Represents the Cth spectrum; focusing on the spectral features, the spectral token captures global information through Mamba's state space model and obtains local information through the window scanning mechanism:

[0127] ;

[0128] ;

[0129] in, To map all the information of each element from different directions from the four corners of the feature map to the relative position, It is a window scanning mechanism that captures local information between sequence elements. Represents the global information gathered by the four-way scanning strategy, Represents the local information gathered by window scanning, and The two are input together into the selective scanning spatial state sequence model and processed in parallel to obtain the intermediate deep spectral feature out_y;

[0130] Step c: Normalize the intermediate deep spectral features and compare Multiply the elements of each part to perform feature weighting, highlight the important features needed, and obtain the intermediate features:

[0131] ;

[0132] in, is the intermediate feature, is a linear layer operation, is the layer normalization operation;

[0133] Step d: Based on the intermediate features, the decoder is used to obtain the deep spectral features. ;

[0134] The sequence of correctly encoded position information is input into the spatial Mamba block, and deep spectral spatial features are obtained through the decoder, specifically including:

[0135] According to the deep spectral characteristics , re-obtain the sequence after correctly encoding the position information;

[0136] Replace the sequence of correctly encoded position information in step a with the sequence of correctly encoded position information that has been re-obtained, and then repeat steps a, b, and c to re-obtain the intermediate features;

[0137] Based on the re-acquired intermediate features, deep spectral spatial features are obtained through the decoder.

[0138] like Figure 3 , which is a schematic diagram of the framework of the Mamba block provided by an embodiment of the present invention; wherein the S6 module is a selective scanning space state sequence model.

[0139] Specifically, the four subband feature maps are sequentially input into the spectral Mamba block and the spatial Mamba block, and the feature maps are expanded to generate a 2D sequence. In order to retain the spatial information of the sequence, position coding is added to the 2D sequence. Finally, the encoded sequence is input into the Mamba network of the two Mamba blocks. The local information sequence and the global information sequence obtained under the cross-scanning mechanism of the Mamba network itself and the introduced window scanning mechanism are selectively scanned in the spatial state sequence model to enhance the feature expression capabilities of the global and local features. The specific operations are as follows: a, b, and c;

[0140] The above operations can obtain deep spectral features when input into the spectral Mamba block. The deep spectral features can be expanded into a 2D sequence by repeating the above operations and then input into the spatial Mamba block to obtain deep spectral spatial features.

[0141] In one embodiment, multi-scale deep spectral spatial features are fused to obtain fused deep spectral spatial features, and spectral spatial features are obtained based on the fused deep spectral spatial features and shallow spectral spatial features;

[0142] The fusing of multi-scale deep spectral spatial features to obtain fused deep spectral spatial features specifically includes:

[0143] The multi-scale deep spectral spatial features are reconstructed into the original signal through the inverse two-dimensional discrete wavelet transform to obtain the fused deep spectral spatial features. The expression is as follows:

[0144] ;

[0145] in, is the deep spectral spatial feature after fusion; is the inverse two-dimensional discrete wavelet transform; Represents the deep spectral spatial features obtained by the cA subband feature map, represents the deep spectral spatial features obtained by the cH subband feature map, Represents the deep spectral spatial features obtained by the cV sub-band feature map, Represents the deep spectral spatial features obtained by the cD sub-band feature map; specifically, is the deep spectral spatial feature obtained by the low-frequency component, is the deep spectral spatial feature obtained by the high-frequency horizontal component, is the deep spectral spatial feature obtained by the high-frequency vertical component, It is the deep spectral spatial feature obtained by the high-frequency diagonal component.

[0146] In one embodiment, obtaining spectral spatial features based on the fused deep spectral spatial features and shallow spectral spatial features specifically includes:

[0147] Use jump connections to aggregate shallow spectral spatial features and fused deep spectral spatial features to obtain spectral spatial features,

[0148] In one embodiment, a high-resolution hyperspectral image is acquired based on the spectral spatial characteristics and the upsampled hyperspectral image.

[0149] The step of obtaining a high-resolution hyperspectral image based on the spectral spatial characteristics and the upsampled hyperspectral image specifically includes:

[0150] Input the spectral spatial features into the image reconstruction module to obtain a high-resolution reconstructed image;

[0151] The high-resolution hyperspectral image and the upsampled hyperspectral image are connected using residuals to fuse low-resolution and high-resolution information to obtain a high-resolution hyperspectral image.

[0152] Inputting the spectral spatial features into the image reconstruction module to obtain a high-resolution reconstructed image specifically includes:

[0153] The spectral spatial features are input into the image reconstruction module to obtain a high-resolution reconstructed image. The specific expression is as follows:

[0154] ;

[0155] ;

[0156] in, is a convolution operation with a convolution kernel of 3×3. To activate the operation; is the addition operation; is the spectral spatial feature; Reconstruct images for high resolution;

[0157] The high-resolution reconstructed image and the upsampled hyperspectral image are connected using residuals to fuse low-resolution and high-resolution information to obtain a high-resolution hyperspectral image. Specifically, the method includes:

[0158] ;

[0159] where Y represents the final high-resolution hyperspectral image, is the input multispectral image, The input hyperspectral image, is the upsampling operation, It is a splicing operation. is the feature extraction operation, is the image reconstruction operation, It is an operation to limit the value to [0, 1].

[0160] like Figure 4-5 As shown in FIG, the RGB heat map of the hyperspectral image of the CAVE dataset and the RGB heat map of the multispectral image of the CAVE dataset provided by the embodiment of the present invention are shown; Figure 6 As shown in FIG, the CAVE dataset fusion result diagram provided by the embodiment of the present invention; In this embodiment, the CAVE dataset is used to generate a low-resolution hyperspectral image as shown in FIG. Figure 4 As shown in (the 10th, 20th, and 30th bands are used as RGB primary colors respectively), the high-resolution multispectral image is as follows Figure 5As shown in the figure; the CAVE dataset contains 32 indoor hyperspectral images, each with a size of 512×512 pixels, covering a wavelength range of 400–700 nm in 31 frequency bands; the existing TFNet (Remote sensing image fusion based on two-stream fusion network), CSSNet (Hyperspectral image super-resolutionnetwork based on cross-scale nonlocal attention), Fusformer (A transformer-based fusion network forhyperspectral image super-resolution), PSRT (Pyramid shuffle-and-reshuffle transformer for multispectral andhyper-spectral image fusion) and the fusion method of the present application are used to fuse hyperspectral images and multispectral images on the CAVE dataset of the embodiment, as shown in the figure. Figure 6 As shown in Table 1, it includes high-resolution hyperspectral images, true label images, absolute error difference maps and structural similarity index maps. The fusion results are shown in Table 1:

[0161] Table 1 Comparison of hyperspectral image and multispectral image fusion results

[0162]

[0163] As can be seen from Table 1, SSIM is the structural similarity index and SAM is the spectral angle mapper; the PSNR (peak signal-to-noise ratio) of the fusion result of the hyperspectral image and multispectral image in this application reached 47.52, which is 0.96 higher than the second-ranked Fusformer method compared with other methods.

[0164] The fusion results of the method applied in the present invention are compared with the real pictures. The first row shows the fusion results and the real pictures, and the second row intuitively shows the results from two dimensions. The figure proves the feasibility of the method applied in the present invention and the good fusion effect.

[0165] The above embodiments demonstrate that the present application reduces the error value of the fusion of hyperspectral images and multispectral images.

[0166] Compared with the existing technology, the present invention uses a method based on Mamba to use wavelet transform to fuse hyperspectral images and multispectral images at multiple scales. First, the hyperspectral image is upsampled, and then the upsampled hyperspectral image and the multispectral image are channel-stitched and fused, and then the shallow features of the fused image are extracted; then, the shallow features are decomposed by wavelet transform to obtain feature maps of multiple scales, and the small feature maps are serialized and input into the Mamba module to obtain deep features; the Mamba module uses a state-space model to efficiently model long sequence data, capture global dependencies, and enhance the global context modeling capability of sequence data through a cross-scanning mechanism; a window scanning mode is introduced to enhance the modeling capability of local features and efficiently extract global and local features; finally, the extracted deep features and shallow features are fused to obtain the final spectral spatial features, and then the features are reconstructed to obtain a high-resolution hyperspectral image.

[0167] The present invention proposes a multi-scale fusion method based on the Mamba model and wavelet transform. The deep feature extraction module (Mamba) can efficiently process long sequence data. The wavelet transform can effectively capture the local features of the image through multi-scale analysis, thereby realizing the multi-scale fusion of hyperspectral images and multispectral images. While improving the spatial resolution, the spectral information is fully retained. The method proposed in the present invention realizes multi-scale feature extraction through wavelet transform, and adds a local feature extraction module to the Mamba module to fully obtain spectral information and spatial information, thereby realizing the multi-scale fusion of hyperspectral images and multispectral images, improving the accuracy of feature extraction, obtaining higher-quality high-resolution hyperspectral images, and solving the problem of low fusion accuracy caused by insufficient retention of spectral information and spatial information in the process of fusing multi-source remote sensing images.

[0168] Example 2:

[0169] like Figure 7 FIG. 1 is a schematic diagram of a structure of a device for multi-scale fusion of hyperspectral images and multispectral images provided by an embodiment of the present invention, wherein the device includes:

[0170] Preliminary fusion module: obtain hyperspectral images and multispectral images; upsample the hyperspectral images to obtain upsampled hyperspectral images; stitch and fuse the upsampled hyperspectral images with the multispectral images to obtain fused hyperspectral images;

[0171] Shallow feature extraction module: extracts features from the fused hyperspectral image to obtain shallow spectral spatial features;

[0172] Deep feature extraction module: performs wavelet transform on shallow spectral spatial features to obtain multi-scale sub-band feature maps; inputs the multi-scale sub-band feature maps into the deep feature extraction module to obtain multi-scale deep spectral spatial features;

[0173] Fusion and reconstruction module: Fusion of multi-scale deep spectral spatial features to obtain fused deep spectral spatial features, and spectral spatial features based on the fused deep spectral spatial features and shallow spectral spatial features; and high-resolution hyperspectral images based on the spectral spatial features and the upsampled hyperspectral images.

[0174] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0178] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for multi-scale fusion of hyperspectral images and multispectral images, characterized in that: The following steps are involved: Acquire hyperspectral and multispectral images; Upsampling the hyperspectral image to obtain an upsampled hyperspectral image, and concatenating and fusing the upsampled hyperspectral image with the multispectral image to obtain a fused hyperspectral image; The fused hyperspectral image is subjected to feature extraction by a shallow feature extraction module to obtain shallow spectral spatial features; wherein the shallow feature extraction module includes convolution and multiple residual blocks, wherein the convolution is used to perform preliminary feature extraction on the input initial features to obtain first extracted features; the multiple residual blocks are used to extract higher-level features layer by layer from the first extracted features to obtain second extracted features, and then the initial features are residually connected with the second extracted features to obtain shallow spectral spatial features; Perform wavelet transform on shallow spectral spatial features to obtain multi-scale sub-band feature maps; The multi-scale sub-band feature map is input into the deep feature extraction module to obtain the multi-scale deep spectral spatial features; Fusing multi-scale deep spectral spatial features to obtain fused deep spectral spatial features, and obtaining spectral spatial features based on the fused deep spectral spatial features and shallow spectral spatial features; Obtain high-resolution hyperspectral images based on spectral spatial characteristics and upsampled hyperspectral images; Wherein, the deep feature extraction module includes a spectral Mamba block and a spatial Mamba block; The multi-scale sub-band feature map is input into the deep feature extraction module to obtain the multi-scale deep spectral spatial feature, which specifically includes performing the following steps on the sub-band feature map of each scale: Step A: Flatten the subband feature map into a vector form and serialize the vector to obtain a sequence; Step B: Generate position codes through the encoder and add the position codes to the sequence, retaining the spatial information of the sequence and obtaining the sequence with the correct encoded position information; Step C: Input the sequence with correctly encoded position information into the spectral Mamba block and obtain deep spectral features through the decoder; The subband feature map in step A is replaced by the deep spectral feature, and then steps A and B are repeated to re-obtain the sequence after the correct encoding position information is obtained. The sequence after the correct encoding position information is input into the spatial Mamba block, and the deep spectral spatial feature is obtained through the decoder.

2. The method for multi-scale fusion of hyperspectral images and multispectral images according to claim 1, characterized in that: The upsampling of the hyperspectral image to obtain the upsampled hyperspectral image specifically includes: Perform bilinear interpolation upsampling on the hyperspectral image to obtain an upsampled hyperspectral image; The step of splicing and fusing the upsampled hyperspectral image with the multispectral image to obtain a fused hyperspectral image specifically includes: The multispectral image is stitched and fused with the upsampled hyperspectral image using a stitching and fusion operation to obtain a fused hyperspectral image.

3. The method for multi-scale fusion of hyperspectral images and multispectral images according to claim 1, characterized in that: The feature extraction is performed on the fused hyperspectral image to obtain the shallow spectral spatial feature, which is expressed as follows: ； in, It is the shallow spectral spatial feature; is the fused hyperspectral image; It is a shallow feature extraction module.

4. The method for multi-scale fusion of hyperspectral images and multispectral images according to claim 1, characterized in that: The wavelet transform of the shallow spectral spatial features to obtain the multi-scale sub-band feature map specifically includes: Select Haar wavelet as the mother wavelet; The input shallow spectral spatial features are low-pass and high-pass filtered in the horizontal and vertical directions respectively to generate a multi-scale sub-band feature map. The multi-scale sub-band feature map includes Subband feature map, Subband feature map, Subband feature map and Subband feature map, where The sub-band feature map represents the low-frequency component; The sub-band feature map represents the high-frequency component in the horizontal direction; The sub-band feature map represents the high-frequency component in the vertical direction; The sub-band feature map represents the high-frequency components in the diagonal direction.

5. The method for multi-scale fusion of hyperspectral images and multispectral images according to claim 4, characterized in that: The sequence after correctly encoding the position information is input into the spectral Mamba block, and the deep spectral features are obtained through the decoder, specifically including: Step a: Sequence after correctly encoding position information Perform linear projection, and the projected features are divided into two feature parts along the channel dimension and , After activation function processing, p performs convolution to extract spatial features and then performs activation function processing to achieve feature block processing and spatial feature enhancement: ； ； ； in, For the projection operation, is the activation function, To correctly encode the sequence of position information, is the convolution operation, R is the scale, B is the batch, H is the height of the multispectral image, W is the width of the multispectral image, and C is the number of channels of the convolution in the shallow feature extraction module; Step b: The feature part p generates a spectrum token M Or generate space token N , focusing on spectral features, the spectral token captures global information through Mamba's state space model and obtains local information through the window scanning mechanism: ； ； in, To map all the information of each element from different directions from the four corners of the feature map to the relative position, It is a window scanning mechanism that captures local information between sequence elements. Represents the global information gathered by the four-way scanning strategy, Represents the local information gathered by window scanning, and The two are input together into the selective scanning spatial state sequence model and processed in parallel to obtain the intermediate deep spectral feature out_y; Step c: Normalize the intermediate deep spectral features and compare Multiply the elements of each part to perform feature weighting, highlight the important features needed, and obtain the intermediate features: ； in, is the intermediate feature, is a linear layer operation, is the layer normalization operation; Step d: Based on the intermediate features, the decoder is used to obtain the deep spectral features. ; The sequence of correctly encoded position information is input into the spatial Mamba block, and deep spectral spatial features are obtained through the decoder, specifically including: According to the deep spectral characteristics , re-obtain the sequence after correctly encoding the position information; Replace the sequence of correctly encoded position information in step a with the sequence of correctly encoded position information that has been re-obtained, and then repeat steps a, b, and c to re-obtain the intermediate features; Based on the re-acquired intermediate features, deep spectral spatial features are obtained through the decoder.

6. The method for multi-scale fusion of hyperspectral images and multispectral images according to claim 4, characterized in that: The fusing of multi-scale deep spectral spatial features to obtain fused deep spectral spatial features specifically includes: The multi-scale deep spectral spatial features are reconstructed into the original signal through the inverse two-dimensional discrete wavelet transform to obtain the fused deep spectral spatial features. The expression is as follows: ； in, is the deep spectral spatial feature after fusion; is the inverse two-dimensional discrete wavelet transform; Represents the deep spectral spatial features obtained by the cA subband feature map, represents the deep spectral spatial features obtained by the cH subband feature map, Represents the deep spectral spatial features obtained by the cV sub-band feature map, Represents the deep spectral spatial features obtained by the cD sub-band feature map.

7. The method for multi-scale fusion of hyperspectral images and multispectral images according to claim 6, characterized in that: The obtaining of spectral spatial features according to the fused deep spectral spatial features and shallow spectral spatial features specifically includes: The spectral spatial features are obtained by using jump connections to aggregate the shallow spectral spatial features and the fused deep spectral spatial features.

8. The method for multi-scale fusion of hyperspectral images and multispectral images according to claim 2, characterized in that: The step of obtaining a high-resolution hyperspectral image based on the spectral spatial characteristics and the upsampled hyperspectral image specifically includes: Input the spectral spatial features into the image reconstruction module to obtain a high-resolution reconstructed image; The high-resolution reconstructed image and the upsampled hyperspectral image are connected using residuals to fuse low-resolution and high-resolution information to obtain a high-resolution hyperspectral image.

9. A device for multi-scale fusion of hyperspectral images and multispectral images, based on the method for multi-scale fusion of hyperspectral images and multispectral images according to any one of claims 1 to 8, characterized in that: The device comprises: Preliminary fusion module: obtain hyperspectral images and multispectral images; upsample the hyperspectral images to obtain upsampled hyperspectral images; stitch and fuse the upsampled hyperspectral images with the multispectral images to obtain fused hyperspectral images; Shallow feature extraction module: extracts features from the fused hyperspectral image to obtain shallow spectral spatial features; Deep feature extraction module: performs wavelet transform on shallow spectral spatial features to obtain multi-scale sub-band feature maps; inputs the multi-scale sub-band feature maps into the deep feature extraction module to obtain multi-scale deep spectral spatial features; Fusion and reconstruction module: Fusion of multi-scale deep spectral spatial features to obtain fused deep spectral spatial features, and spectral spatial features based on the fused deep spectral spatial features and shallow spectral spatial features; and high-resolution hyperspectral images based on the spectral spatial features and the upsampled hyperspectral images.

Images