A multi-level collaborative mapping method for fusion of hyperspectral and multispectral remote sensing images

Abstract

The present invention belongs to the field of image processing technology, and relates to a multi-level collaborative mapping hyperspectral and multispectral remote sensing image fusion method, including a latent space projection unit, a multi-level feature mapping unit and an adaptive feature fusion unit; the latent space projection unit projects the input image data into a latent representation space through two convolutional layers, and this process is completed by a mapping function obtained by learning; the multi-level feature mapping unit then operates in the latent space, which is divided into three levels: point-level, local-level and global-level mapping; the point-level mapping focuses on feature extraction at the pixel level, while the local-level mapping focuses on feature interaction within the neighborhood, and the global-level mapping focuses on contextual information at the entire image level. The adaptive feature fusion unit integrates multi-level features in a dynamic manner, and realizes information complementarity between hyperspectral and multispectral remote sensing images. The present invention has a good degree of lightweight by introducing Mamba and implicit neural representation architecture.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a multi-level collaborative mapping hyperspectral and multispectral remote sensing image fusion method. Background Art

[0002] The spectral range of hyperspectral remote sensing images (HSI) covers the spectrum from ultraviolet to infrared, and due to its high spectral resolution characteristics, it has been widely used in environmental monitoring, land classification, target detection and other fields. Limited by the physical characteristics of the sensor, when the sensor focuses on the expression of hyperspectral information, it often weakens the spatial resolution at the expense of weakening the spatial resolution, resulting in the spatial resolution of hyperspectral images being inferior to that of multispectral images (MSI) under fixed input photon energy. Although MSI has a lower spectral resolution, it can more finely depict the morphology and layout of ground cover. In practical applications, there is often a simultaneous demand for high spatial and high spectral resolution. In order to solve the contradiction between the spatial and spectral expressions of remote sensing images, directly improving the performance of sensor hardware is a direct way, but it is accompanied by high costs. In contrast, image fusion technology uses the advantages of high spectral and high spatial resolution of heterogeneous data to generate high spatial and high spectral resolution images, which is an economical and effective method.

[0003] The fusion of HSI and MSI includes traditional methods and deep learning methods. Deep learning methods are increasingly widely used in HMIF and show significant advantages over traditional methods. Deep learning methods rely on the perception of local and global features of convolutional neural networks (CNNs) and Transformer architecture models to achieve breakthrough performance in HMIF tasks. However, most algorithms often overlook a key level of detail, namely point-level mapping. In fact, point-level mapping can provide highly refined features, which is crucial for understanding subtle differences in complex scenes. Current fusion algorithms mainly focus on optimizing the integration of local and global features, but fail to fully explore the synergy between point-level features and these features, limiting the model's expression of complex details. In addition, existing methods tend to adopt discrete processing, which is different from the continuous expression of real space, and discretization processing may lead to the loss of subtle features when processing scenes with complex structures. Finally, current fusion techniques, especially those driven by Transformer, have the problem of high model complexity. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-level collaborative mapping hyperspectral and multispectral remote sensing image fusion method to solve the problem of high model complexity in existing fusion technologies, especially methods driven by Transformer as the core.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A multi-level collaborative mapping hyperspectral and multispectral remote sensing image fusion method includes the following steps:

[0007] S1. Use HSI as the reference label image, perform spatial and spectral downsampling on HSI and MSI, use the processed images as training samples, and the un-downsampled hyperspectral images as reference samples to generate a data set suitable for network parameter adjustment.

[0008] S2. Construct a multi-level collaborative mapping fusion network model, which consists of a latent space projection unit, a multi-level collaborative mapping unit, and an adaptive fusion unit. The latent space projection unit is based on convolutional neural network technology. The multi-level collaborative mapping unit integrates three sub-modules: point mapping, local mapping, and global mapping. Point mapping is implemented through an implicit neural network. Local mapping first divides image features into regions, and then uses the Vision Mamba algorithm to map local features. Global mapping directly applies the Vision Mamba algorithm to map overall features.

[0009] S3, using the Adam optimization algorithm to systematically train the multi-level collaborative mapping fusion network to adjust network parameters and obtain a trained network model;

[0010] S4. Input the HSI and MSI to be fused into the fusion network trained in step S3 to generate a remote sensing image with both high spatial and high spectral resolution.

[0011] Further, step S1 is specifically as follows: according to the Wald protocol, the HSI and MSI are first Gaussian filtered respectively; then the bilinear interpolation method is used to downsample them by corresponding multiples to simulate the sensor point spread and spectral response function, so that the processed images are used as HSI and MSI to simulate low-resolution input, and the original HSI is retained as a reference image.

[0012] Furthermore, in step S2, the latent space projection unit shallowly encodes HSI and MSI through parallel convolution layers, projects the image into the latent space, uses 3×3 convolution kernels to scan the input data pixel by pixel, and sets the number of convolution kernels to 64. While capturing local features in the image, HSI and MSI are processed in parallel to achieve alignment of different spectral information, which can be expressed as the following formula:

[0013] (1)

[0014] (2)

[0015] (3)

[0016] in, and They are HSI and MSI, and They are the convolutional layers for potential projection, and They are the HSI and MSI after projection, represents the joint features prepared for local and global feature projection, represents the feature concatenation operation, Convolution operation for spectral dimension reduction.

[0017] Furthermore, in step S2, the point mapping unit captures fine features in the image through a point-to-point function, and uses the projected MSI ( )’s high spatial detail guided projection HSI ( ) for airspace enhancement; and Mapping to a coordinate grid To achieve spatial alignment, the coordinates are scaled to the range (-1, 1) to uniformly represent the features of MSI and HSI in a shared coordinate system. For each interpolation target domain interpolation point , whose input includes The 4 nearby Pixels, corresponding pixels, and Pixels relative to these four The relative coordinates of the pixels are nonlinearly mapped through the multi-layer perceptron (MLP) to calculate the interpolation of the corresponding pixels; the softmax layer calculates each Pixel Pair The interpolation weight of is used to realize the mapping of the point in a weighted manner, which is expressed by the following formula:

[0018] (4)

[0019] (5)

[0020] (6)

[0021] in, , They are The corner pixel of the query point The interpolation contribution of is the vector of the query point pixel, and the point mapping feature is obtained by traversing all pixels .

[0022] Furthermore, in step S2, the local mapping unit first divides the input features into multiple local patches in a non-overlapping manner. Each window is regarded as an independent image block. Vision Mamba is used to perform local feature mapping on each patch to extract its internal key features and patterns. Vision Mamba uses its efficient linear complexity architecture and multi-resolution feature aggregation capabilities to merge all mapped local features into a complete global feature map through splicing or fusion operations, which is expressed as the following formula:

[0023] (7)

[0024] (8)

[0025] (9)

[0026] Among them, Partition represents the window partitioning operation, which divides the input features into multiple local patches; After segmentation Patch, Vmamba uses Vision Mamba to extract features. For the The mapping feature of each patch, Merge is the feature merging operation, It is the feature output by the local mapping unit, and the Patch size is 16x16.

[0027] Furthermore, in step S2, the global mapping unit directly uses the global perception capability of VisionMamba to perform feature mapping by integrating global information; the global mapping unit also uses the linear complexity characteristics of Vision Mamba to ensure that the computational efficiency is not affected when processing large-scale features, which is expressed by the following formula:

[0028] (10)

[0029] in, It is the feature output by the global mapping unit.

[0030] Furthermore, the point-level mapping features are used as basic features, while the local and global mapping features are used as supplementary features for detail enhancement. After the local and global features are normalized by the sigmoid activation function, they are respectively element-wise multiplied with the point-level mapping features. The results are injected into the point-level mapping features as supplementary details to form the final adaptive fusion features. The spectral dimension is restored to the same HSI through convolution mapping to generate HR-HIS, which is expressed as:

[0031] (11)

[0032] (12)

[0033] Further, step S3 is specifically as follows: when the Adam optimization algorithm is used to train the multi-level collaborative mapping fusion network, the loss function uses the L1 loss function to constrain the network, which guides the network optimization direction by measuring the absolute difference error between the fused image and the reference image, and is expressed by the following formula:

[0034] (13)

[0035] in, is the fused image of the current state output by the network, is the reference image, and N is the total number of pixels in the image.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention provides a method for fusing hyperspectral and multispectral remote sensing images with point, local and global collaborative mapping, the method comprising a latent space projection unit, a multi-level feature mapping unit and an adaptive feature fusion unit; the space projection unit projects input image data into a latent representation space through two convolutional layers, and this process is completed by a mapping function obtained by learning; the multi-level feature mapping unit then operates in the latent space, and is divided into three levels of point-level, local-level and global-level mapping; the point-level mapping focuses on feature extraction at the pixel level, while the local-level mapping focuses on feature interaction within the neighborhood, and the global-level mapping focuses on contextual information at the entire image level; the adaptive feature fusion unit integrates multi-level features in a dynamic manner, and realizes information complementarity between hyperspectral and multispectral remote sensing images; in addition, the present invention has a good degree of lightweight by introducing Mamba and implicit neural representation architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 A flowchart of the steps of a multi-level collaborative mapping hyperspectral and multispectral remote sensing image fusion method provided by the present invention;

[0040] Figure 2A network structure diagram of a multi-level collaborative mapping hyperspectral and multispectral remote sensing image fusion method provided by the present invention;

[0041] Figure 3 This is a diagram showing the fusion results of different hyperspectral and multispectral fusion algorithms in the present invention under the ITRES CASI 1500 sensor. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with embodiments:

[0043] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0044] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0045] like Figure 1 As shown, a multi-level collaborative mapping hyperspectral and multispectral remote sensing image fusion method includes the following steps:

[0046] S1. In actual operation, due to the lack of fused images as training references, the present invention follows the sample construction technology described in the Wald protocol. The specific steps include: first, using a hyperspectral remote sensing image (HSI) as a reference label image. Then, performing spatial and spectral downsampling processing on the hyperspectral image and multispectral remote sensing image (MSI) as training samples, and using the un-downsampled hyperspectral image as a reference sample to generate a data set suitable for network parameter adjustment.

[0047] S2, such as Figure 2 As shown in the figure, a multi-level collaborative mapping fusion network model is constructed, which consists of three main parts: spatial projection unit, multi-level collaborative mapping unit and adaptive fusion unit. Among them, the spatial projection unit is based on convolutional neural network technology; the multi-level collaborative mapping unit integrates three sub-modules: point mapping, local mapping and global mapping. Point mapping is realized through implicit neural network. Local mapping first divides the image features into regions, and then uses VisionMamba algorithm to map local features, while global mapping directly applies Vision Mamba algorithm to map the overall features.

[0048] S3. The Adam optimization algorithm is used to systematically train the multi-level collaborative mapping fusion network. Through this process, the network parameters are adjusted to finally obtain a fully trained network model.

[0049] S4. Input the HSI and MSI to be fused into the fusion network trained in step 3 to generate a remote sensing image with both high spatial and high spectral resolution.

[0050] Step S1 specifically includes the following steps:

[0051] According to the Wald protocol, first, the HSI and MSI are Gaussian filtered; then, they are downsampled by the corresponding multiples using bilinear interpolation, which simulates the effect of sensor point spread and spectral response function. The processed images are used as HSI and MSI to simulate low-resolution input, and the original HSI is retained as a reference image.

[0052] Step S2 specifically includes the following steps:

[0053] The latent space projection unit shallowly encodes HSI and MSI through parallel convolution layers, aiming to project these images into the latent space and lay the foundation for subsequent multi-level feature mapping. Specifically, the present invention uses a 3×3 convolution kernel to scan the input data pixel by pixel, and sets the number of convolution kernels to 64, which meets the projection accuracy while maintaining efficient calculation. This design can not only effectively capture local features in the image, but also align different spectral information by processing HSI and MSI in parallel. The mathematical expression is:

[0054] (1)

[0055] (2)

[0056] (3)

[0057] in, and They are HSI and MSI, and They are the convolutional layers for potential projection, and They are the HSI and MSI after projection, represents the joint features prepared for local and global feature projection, represents the feature concatenation operation, The convolution operation is used for spectral dimension reduction. In this way, the latent space projection unit provides the feature representation required for subsequent multi-level feature mapping.

[0058] The multi-level collaborative mapping unit includes point, local and global mapping modules. Each module is introduced next.

[0059] The point mapping unit captures fine features in the image through a point-to-point function and uses the projected MSI ( )’s high spatial detail guided projection HSI ( ) for airspace enhancement; and Mapping to a coordinate grid To achieve spatial alignment, the coordinates are scaled to the range (-1, 1) to uniformly represent the features of MSI and HSI in a shared coordinate system. For each interpolation target domain interpolation point , whose input includes The 4 nearby Pixels, corresponding pixels, and Pixels relative to these four The relative coordinates of the pixels are nonlinearly mapped through the multi-layer perceptron (MLP) to calculate the interpolation of the corresponding pixels; the softmax layer calculates each Pixel Pair The interpolation weight of is used to realize the mapping of the point in a weighted manner, which is expressed by the following formula:

[0060] (4)

[0061] (5)

[0062] (6)

[0063] in, , They are The corner pixel of the query point The interpolation contribution of is the vector of the query point pixel, and the point mapping feature is obtained by traversing all pixels .

[0064] The local mapping unit first divides the input features into multiple local patches in a non-overlapping manner. Each window is regarded as an independent image block. Vision Mamba is used to perform local feature mapping on each patch to extract its internal key features and patterns. Vision Mamba uses its efficient linear complexity architecture and multi-resolution feature aggregation capabilities to merge all mapped local features into a complete global feature map through splicing or fusion operations, which is expressed as the following formula:

[0065] (7)

[0066] (8)

[0067] (9)

[0068] Among them, Partition represents the window partitioning operation, which divides the input features into multiple local patches; After segmentation Patch, Vmamba uses Vision Mamba to extract features. For the The mapping feature of each patch, Merge is the feature merging operation, It is the feature output by the local mapping unit, and the Patch size is 16x16.

[0069] The global mapping unit directly uses the global perception capability of Vision Mamba to perform feature mapping by integrating global information. The global mapping unit also uses the linear complexity characteristics of Vision Mamba to ensure that the computational efficiency is not affected when processing large-scale features, which is expressed as the following formula:

[0070] (10)

[0071] in, It is the feature output by the global mapping unit.

[0072] The adaptive feature fusion unit aims to effectively integrate point-level, local and global mapping features to improve the fusion quality. Due to the strong correlation between point, local and global mapping features, the adaptive fusion unit designed in the present invention first sets one feature as the basic feature and the remaining features as supplementary features. It was found in the experiment that the point-level mapping feature played a key role in the overall fusion effect, which will be demonstrated in detail in the subsequent ablation experiment. Therefore, the point-level mapping feature is used as the basic feature, while the local and global mapping features are used as supplementary features for detail enhancement. Specifically, the local and global features are normalized by the sigmoid activation function, and then element-wise multiplied with the point-level mapping features respectively. The results are injected into the point-level mapping features as supplementary details to form the final adaptive fusion features. This method achieves effective fusion and enhancement of features by adaptively adjusting the contribution of each feature. Finally, the spectral dimension is restored to the same HSI through convolution mapping to generate HR-HSI. The formula is expressed as:

[0073] (11)

[0074] (12)

[0075] Step S3 specifically includes the following steps:

[0076] When the Adam optimization algorithm is used to train the multi-level collaborative mapping fusion network, the loss function uses the L1 loss function to constrain the network, which guides the network optimization direction by measuring the absolute difference error between the fused image and the reference image. Its expression is:

[0077] (13)

[0078] in, is the fused image of the current state output by the network, is the reference image, and N is the total number of pixels in the image.

[0079] In order to prove the superiority of the fusion effect of the present invention, that is, to prove the fidelity of the spatial and spectral information of the fusion effect of the present invention, the images taken by the ROSIS sensor were subjected to fusion experiments with 4 times, 8 times and 16 times the spatial resolution difference, and the new fusion method at this stage was used for comparison. The results are shown in Figure 2. Figure 3 As shown, Figure 3 The first row is the fusion image, the second row represents the spectral angle mapping (SAM) error heat map, the third row is the mean relative absolute error (MRAE) error heat map, and the fourth row is the difference map. Among them, the SAM error heat map reflects the spectral fidelity of the fused image, the MRAE error heat map reflects the degree of global consistency of the fused image, and the difference map (DIF) measures the residual of individual bands. The darker the color, the better the fusion result. It can be clearly seen that the present invention has the best retention of spatial and spectral detail information. The quantitative evaluation of its 4x, 8x and 16x fusion is shown in Table 1, and a complexity comparison analysis is carried out with the current deep algorithm, see Table 2, which once again shows that the present invention has certain advantages and lightweight.

[0080] Table 1 Quantitative evaluation of fusion experiments of the present invention and the comparison algorithm

[0081]

[0082] Table 2 Comparison of complexity between the present invention and the comparative algorithm

[0083]

[0084] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A multi-level collaborative mapping method for fusion of hyperspectral and multispectral remote sensing images, characterized in that: The following steps are involved: S1. Use HSI as the reference label image, perform spatial and spectral downsampling on HSI and MSI, use the processed images as training samples, and the un-downsampled hyperspectral images as reference samples to generate a data set suitable for network parameter adjustment. S2. Construct a multi-level collaborative mapping fusion network model, which consists of a latent space projection unit, a multi-level collaborative mapping unit, and an adaptive fusion unit. The latent space projection unit is based on convolutional neural network technology. The multi-level collaborative mapping unit integrates three sub-modules: point mapping, local mapping, and global mapping. Point mapping is implemented through an implicit neural network. Local mapping first divides image features into regions, and then uses the Vision Mamba algorithm to map local features. Global mapping directly applies the Vision Mamba algorithm to map overall features. Among them, the latent space projection unit shallowly encodes HSI and MSI through parallel convolution layers, projects the image into the latent space, uses 3×3 convolution kernels to scan the input data pixel by pixel, and sets the number of convolution kernels to 64. While capturing the local features in the image, the HSI and MSI are processed in parallel to achieve alignment of different spectral information, which can be expressed as the following formula: in, and HSI and MSI, respectively. φ and They are the convolutional layers for potential projection, and They are the HSI and MSI after projection, represents the joint features prepared for local and global feature projection, Cat represents the feature concatenation operation, and F β It is used for spectral dimension reduction convolution operation; The point mapping unit captures fine features in the image through point-to-point functions and uses the projected MSI High spatial detail guides the projection Conduct airspace enhancement; and Mapping to a coordinate grid To achieve spatial alignment, the coordinates are scaled to the range (-1, 1) to uniformly represent the features of MSI and HSI in a shared coordinate system. For each interpolation target domain interpolation point x q , whose input includes x q The 4 nearby Pixels, corresponding Pixels, and x q Pixels relative to these four The relative coordinates of the pixels are mapped nonlinearly through a multi-layer perceptron (MLP) to calculate the interpolation of the corresponding pixels; the softmax layer calculates the Pixel pair x q The interpolation weight of is used to realize the mapping of the point in a weighted manner, which is expressed by the following formula: in, are the i-th corner pixel for the query point x q The interpolation contribution of is the vector of the query point pixel, and the point mapping feature is obtained by traversing all pixels The local mapping unit first divides the input features into multiple local patches in a non-overlapping manner. Each window is regarded as an independent image block. Vision Mamba is used to perform local feature mapping on each patch to extract its internal key features and patterns. Vision Mamba uses its efficient linear complexity architecture and multi-resolution feature aggregation capabilities to merge all mapped local features into a complete global feature map through splicing or fusion operations, which is expressed as the following formula: Among them, Partition represents the window partitioning operation, which divides the input features into multiple local patches; i is the ith Patch after segmentation. Vmamba uses Vision Mamba to extract features. is the mapping feature of the ith Patch, Merge is the feature merging operation, and F L The features output by the local mapping unit use a Patch size of 16x16; The global mapping unit directly uses the global perception capability of Vision Mamba to perform feature mapping by integrating global information. The global mapping unit also uses the linear complexity characteristics of Vision Mamba to ensure that the computational efficiency is not affected when processing large-scale features, which is expressed as the following formula: Among them, F G Features output by the global mapping unit; The point-level mapping features are used as basic features, while the local and global mapping features are used as supplementary features for detail enhancement. After the local and global features are normalized by the sigmoid activation function, they are respectively multiplied element-wise with the point-level mapping features. The results are injected into the point-level mapping features as supplementary details to form the final adaptive fusion features. The spectral dimension is restored to the same HSI through convolution mapping to generate HR-HIS, which is expressed by the following formula: F AFF =F P +F P *sigmoid(F L )++F P *sigmoid(F G ) (11) x=F θ (F AFF ) (12); S3, using the Adam optimization algorithm to systematically train the multi-level collaborative mapping fusion network to adjust network parameters and obtain a trained network model; S4. Input the HSI and MSI to be fused into the fusion network trained in step S3 to generate a remote sensing image with both high spatial and high spectral resolution.

2. The method for fusion of hyperspectral and multispectral remote sensing images by multi-level collaborative mapping according to claim 1, characterized in that: Step S1, specifically, is as follows: according to the Wald protocol, the HSI and MSI are first subjected to Gaussian filtering respectively; then, the bilinear interpolation method is used to downsample them by corresponding multiples to simulate the sensor point spread and spectral response function, so that the processed images are used as HSI and MSI to simulate low-resolution input, and the original HSI is retained as a reference image.

3. The method for fusion of hyperspectral and multispectral remote sensing images by multi-level collaborative mapping according to claim 1, characterized in that: Step S3, specifically: when the Adam optimization algorithm is used to train the multi-level collaborative mapping fusion network, the loss function uses the L1 loss function to constrain the network, which guides the network optimization direction by measuring the absolute difference error between the fused image and the reference image, and is expressed by the following formula: Among them, fusion is the fused image of the current state output by the network, reference is the reference image, and N is the total number of pixels in the image.

Images