A hyperspectral and multispectral image fusion method, system and storage medium based on spectral state fusion tree Mamba

Abstract

The present invention discloses a hyperspectral and multispectral image fusion method, system, and storage medium based on a spectral state fusion tree Mamba, relating to the field of computer vision technology. The method comprises obtaining a hyperspectral image and a multispectral image; taking the hyperspectral image and the multispectral image as input, and outputting a super-resolution fused reconstructed image based on a spectral state fusion tree Mamba network model; the spectral state fusion tree Mamba network model comprises a convolution module, a tree construction module, and a tree scanning module connected in sequence, with an upsampling module connected to the input end of the convolution module. The present invention utilizes a tree Mamba scanning mechanism that adaptively scans the spatial structure of hyperspectral image data to achieve efficient hyperspectral spatial information interaction, overcoming the problem of independent channels and lack of information interaction in the Mamba model. The resulting super-resolution fused reconstructed image maintains low time complexity while achieving superior classification accuracy to existing image fusion algorithms.

Description

Technical Field

[0001] The present invention relates to a hyperspectral and multispectral image fusion method, system and storage medium based on a spectral state fusion tree Mamba, and belongs to the technical field of computer vision. Background Art

[0002] Hyperspectral and multispectral image fusion super-resolution reconstruction technology is committed to combining the nanometer-level spectral resolution of hyperspectral images (HSI) with the sub-meter spatial resolution of multispectral images (MSI) to generate fused images with both wide-band coverage and high spatial details, so as to meet the needs of fine identification and quantitative analysis of ground objects in scenarios such as satellite remote sensing and environmental monitoring.

[0003] However, hyperspectral data is usually limited by sensor hardware and has a low spatial resolution (such as 30 meters / pixel). Although multispectral data has a higher spatial resolution (1~5 meters / pixel), the number of its spectral channels is limited (only 4-8 bands). There is a natural spectral-spatial resolution inversion contradiction between the two.

[0004] While convolutional neural network (CNN)-based methods can extract spatial texture features through local convolution kernels, their limited receptive field makes it difficult to capture long-range spectral correlations across hundreds of bands. Transformer-based models utilize self-attention mechanisms to enhance global modeling capabilities, but their quadratic computational complexity leads to a surge in memory overhead when processing large-scale remote sensing imagery. The emerging Mamba architecture achieves global modeling with linear complexity through a state-space model (SSM), but its fixed row and column scanning path cannot adapt to the irregular spatial distribution of objects in hyperspectral imagery, and its channel-independent processing mechanism isolates the interaction of spectral features across bands.

[0005] Therefore, existing methods find it difficult to achieve an effective balance among computational efficiency, spectral fidelity, and spatial detail preservation, which becomes a core obstacle to the practical application of hyperspectral and multispectral fusion technology. Summary of the Invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a hyperspectral and multispectral image fusion method, system, and storage medium based on a spectral state fusion tree Mamba. The method utilizes a tree Mamba scanning mechanism that adaptively scans the spatial structure of hyperspectral image data to achieve efficient hyperspectral spatial information interaction. Spectral state fusion overcomes the problem of independent channels and lack of information interaction in the Mamba model. The resulting super-resolution fused reconstructed image maintains low time complexity while achieving better classification accuracy than existing image fusion algorithms.

[0007] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0008] In a first aspect, the present invention provides a hyperspectral and multispectral image fusion method based on a spectral state fusion tree Mamba, comprising:

[0009] Acquire hyperspectral and multispectral images;

[0010] Taking hyperspectral images and multispectral images as input, the Mamba network model based on spectral state fusion tree is:

[0011] Perform spatial upsampling on the hyperspectral image to obtain a hyperspectral image with high spatial resolution;

[0012] The high spatial resolution hyperspectral image and the multispectral image are spliced ​​along the channel dimension to obtain a fused image;

[0013] Perform convolution operation on the fused image to obtain the fused embedded features;

[0014] Generate a minimum spanning tree based on the fused embedded features;

[0015] The minimum spanning tree is used to aggregate information of all pixels in the fused embedded features to obtain the spectral fusion state space vector, and the spectral fusion state space vector is fused with the fused embedded features to obtain the super-resolution fused reconstructed image;

[0016] The spectral state fusion tree Mamba network model includes a convolution module, a tree construction module and a tree scanning module connected in sequence, and the input end of the convolution module is connected to an upsampling module.

[0017] Furthermore, the expression of the hyperspectral image is: , the expression of the high spatial resolution hyperspectral image is , the expression of the multispectral image is , the expression of the fused image is , the expression of the fusion embedding feature is ;

[0018] in, represents the original hyperspectral image, represents a hyperspectral image with high spatial resolution, represents a multispectral image, represents the fused image, Indicates the image height and width in low spatial resolution state, Indicates the image height and width in high spatial resolution state, Indicates the number of channels in low spectral resolution state, Indicates the number of channels in high spectral resolution state, represents the fusion embedding feature, Indicates the number of fused embedding feature channels.

[0019] Furthermore, the data processing process of generating a minimum spanning tree based on the fused embedded features specifically includes:

[0020] For each pixel point in the fusion embedded feature, the corresponding A matrix, B matrix, C matrix, D matrix, and Δ matrix are generated by mapping, and the A matrix, B matrix, C matrix, and D matrix are discretized to obtain the corresponding discretization matrix;

[0021] Calculate the cosine distance between each pixel point in the fused embedded feature and the pixels in the four directions of up, down, left, and right with it as the midpoint;

[0022] The cosine distance between two pixels is used as the weight value of its weighted edge to obtain the feature construction graph;

[0023] The Borufka algorithm is used to prune the feature construction graph to obtain the minimum spanning tree.

[0024] Furthermore, the expressions of the A matrix, B matrix, C matrix, D matrix, and Δ matrix are respectively:

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] in, The coordinates are A matrix of pixels, The coordinates are The B matrix of the pixel points, The coordinates are The C matrix of pixels, The coordinates are The D matrix of pixels, The coordinates are The Δ matrix of pixels, The coordinates are The fusion embedding features of the pixel points, 、 、 、 、 are mapping functions, represents the state space dimension, represents the number of fusion embedding feature channels, Indicates the operation of constructing a diagonal matrix;

[0031] The discretization matrix expressions of the A matrix, B matrix, C matrix, and D matrix are respectively:

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] in, 、 、 、 They are matrix, matrix, matrix, The discretization matrix of the matrix, represents the identity matrix;

[0037] The expressions of the discretization matrices of each pixel in the fused embedded features are:

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] in, The coordinates are Pixels The discretization matrix of the matrix, The coordinates are Pixels The discretization matrix of the matrix, The coordinates are Pixels The discretization matrix of the matrix, The coordinates are Pixels The discretization matrix of the matrix.

[0043] Furthermore, the expression for calculating the cosine distance between each pixel point in the fusion embedded feature and the pixel points in the four directions of up, down, left, and right with the pixel point as the midpoint is:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] in, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points;

[0049] The expression of the characteristic structure graph is: ,in, Represents the feature structure diagram, Represents the pixel set of the feature construction map, A weighted edge set representing a feature construction graph.

[0050] Furthermore, the method of using the minimum spanning tree to aggregate information of all pixels in the fused embedded features to obtain a spectral fusion state space vector, and fusing the spectral fusion state space vector with the fused embedded features to obtain a super-resolution fused reconstructed image includes:

[0051] For each pixel in the fused embedding feature:

[0052] The tree scanning algorithm is used to perform preliminary aggregation of the state space vector of the pixel points in the fused embedded features to obtain the updated state space vector of the pixel points;

[0053] Perform multiple one-dimensional dilated convolutions on the updated state space vector of the pixel to obtain multiple corresponding outputs, and then perform weighted summation of the multiple corresponding outputs to obtain the spectral fusion state space vector of each pixel.

[0054] The discretized matrix of C matrix and D matrix is ​​used to process the spectral fusion state space vector and fusion embedding feature of the pixel point to obtain the output feature of the pixel point;

[0055] The output features of all pixels in the embedded features are combined to obtain a super-resolution fused reconstructed image.

[0056] Furthermore, the method of performing preliminary aggregation of the state space vectors of the pixels using the tree scanning algorithm to obtain the updated state space vectors of the pixels includes:

[0057] The pixel point is used as the root node of the minimum spanning tree to determine multiple leaf nodes, and multiple directed paths from the leaf nodes to the root node are obtained based on the root node and the leaf nodes;

[0058] The discretized matrices of matrix A and matrix B are used to selectively scan and aggregate each directed path to obtain the corresponding directed path information, and all directed path information is aggregated to the state space vector of the root node to obtain the updated state space vector of the pixel point, which is expressed as follows:

[0059] ;

[0060] in, Represents the updated state space vector obtained by taking the pixel point as the root node R, Indicates that Leaf nodes Directed path to the root node R, Indicates that the leaf node Nodes on the directed path to the root node R, Represented by the node Directed path to the root node R, Represented by the node Nodes on the directed path to the root node R, Representation node The A matrix corresponding to the pixel point is discretized matrix, Representation node The B matrix corresponding to the pixel point is discretized matrix, Representation node The fused embedding features of the corresponding pixels, Indicates the number of leaf nodes.

[0061] Furthermore, the expression of the output feature of the pixel point obtained by processing the spectral fusion state space vector and the fusion embedded feature of the pixel point using the discretization matrix of the C matrix and the D matrix is:

[0062] ;

[0063] ;

[0064] in, The coordinates are The output features of the pixel points, The coordinates are The C matrix of the pixel points is discretized into a matrix, The coordinates are The spectral fusion state space vector of the pixel point is, The coordinates are The D matrix of the pixel points is discretized into a matrix, The coordinates are The fusion embedding features of the pixel points, represents the mapping function, represents the convolution operation, Indicates intermediate operation parameters.

[0065] In a second aspect, the present invention further provides a hyperspectral and multispectral image fusion system based on a spectral state fusion tree Mamba, comprising:

[0066] An image acquisition module configured to acquire hyperspectral images and multispectral images;

[0067] The hyperspectral and multispectral image fusion module is configured to take the hyperspectral image and the multispectral image as input, and obtain the super-resolution fused reconstructed image based on the output of the spectral state fusion tree Mamba network model.

[0068] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba as described in any one of the first aspects is implemented.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] The present invention utilizes a tree Mamba scanning mechanism that adaptively scans the spatial structure of hyperspectral image data to achieve efficient hyperspectral spatial information interaction. The state space vector obtained by aggregating the tree scanning information is then used to achieve spectral state fusion using a one-dimensional dilated convolution based on the channel dimension, overcoming the problem of independent channels and lack of information interaction in the Mamba model. Finally, the obtained spectral fusion state space vector is fused with the input fusion embedding feature to generate a super-resolution fused reconstructed image. This method achieves superior classification accuracy to existing image fusion algorithms while maintaining low time complexity.

[0071] The present invention proposes a tree Mamba scanning method, which scans the two-dimensional image in a scanning order according to the structure of a minimum spanning tree. This method can arrange pixels with high spatial and spectral similarity at relatively close positions in the scanning sequence, and arrange pixels with low spatial and spectral similarity at relatively far positions in the scanning sequence, thereby rationally constructing the distance relationship between the pixels in the scanning sequence. Furthermore, the minimum spanning tree can adapt to the spatial structural distribution of the pixels, facilitating the modeling of the Mamba model on two-dimensional image data. Furthermore, the method performs one-dimensional convolution along the spectral dimension on the intermediate state space vector of the scan, thereby realizing spectral information fusion and further fully utilizing the rich physical information in the spectral dimension. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 1. A schematic flow chart of a hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba in one embodiment of the present invention;

[0073] Figure 2 A schematic diagram of the structure of a spectrum state fusion tree Mamba network model in an embodiment of the present invention;

[0074] Figure 3 A schematic diagram of the data processing process of the tree construction module of the spectral state fusion tree Mamba network model in one embodiment of the present invention;

[0075] Figure 4 Schematic diagram of the data processing process of the tree scanning module of the spectral state fusion tree Mamba network model in one embodiment of the present invention;

[0076] Figure 5 A schematic diagram of preliminary aggregation of state space vectors in a tree scanning module of a spectral state fusion tree Mamba network model in an embodiment of the present invention;

[0077] Figure 6 Schematic diagram of one-dimensional dilated convolution processing in a tree scanning module of a spectral state fusion tree Mamba network model in an embodiment of the present invention;

[0078] Figure 7Schematic diagram showing a comparison between a super-resolution fused reconstructed image of a test data "oil painting" using a hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba and output images from other models on a CAVE dataset in one embodiment of the present invention;

[0079] Figure 8 Schematic diagram showing a comparison between a super-resolution fused reconstructed image of the test data "real and fake peppers" and output images of other models using a hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba on a CAVE dataset in one embodiment of the present invention;

[0080] Figure 9 This is a schematic diagram showing a comparison between a super-resolution fused reconstructed image of the test data "plush toys" on the CAVE dataset using a hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba in an embodiment of the present invention and images output by other models. DETAILED DESCRIPTION

[0081] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0082] Example 1:

[0083] The embodiment of the present invention provides a hyperspectral and multispectral image fusion method based on a spectral state fusion tree Mamba, comprising the following steps:

[0084] First, hyperspectral images and multispectral images are acquired.

[0085] The expression of hyperspectral image is: , the expression of multispectral image is ,in, represents the original hyperspectral image, represents a multispectral image, Indicates the image height and width in high spatial resolution state, Indicates the number of channels in low spectral resolution state, Indicates the number of channels in high spectral resolution state.

[0086] Taking hyperspectral images and multispectral images as input, a high-resolution fused reconstructed image is output based on the spectral state fusion tree Mamba network model.

[0087] The structure of the spectral state fusion tree Mamba network model is as follows Figure 2As shown, it includes a convolution module, a tree construction module and a tree scanning module connected in sequence, and the input end of the convolution module is also connected to the output end of the upsampling module.

[0088] like Figure 1 As shown, the hyperspectral image is input from the input end of the upsampling module, and the upsampling module spatially upsamples the hyperspectral image to obtain a hyperspectral image with high spatial resolution, which is expressed as , represents a hyperspectral image with high spatial resolution, Indicates the image height and width in high spatial resolution state, Indicates the number of channels in high spectral resolution state.

[0089] The high spatial resolution hyperspectral image and multispectral image are spliced ​​along the channel dimension to obtain a fused image. The fused image is input into the convolution module for convolution operation to obtain the fused embedded feature. In this embodiment, the convolution kernel size is 3 and the number of convolution kernels is 64. The specific operation can be expressed as follows:

[0090] ;

[0091] in, represents the fusion embedding feature, , ,in, The coordinates are The fusion embedding features of the pixel points, Indicates the image height and width in high spatial resolution state, represents the number of fusion embedding feature channels, represents the convolution operation, represents the fused image, .

[0092] Combine Figure 3 ,The tree construction module is used to generate a minimum spanning tree based on the fused embedding features, specifically including:

[0093] For each pixel in the fusion embedded feature, the corresponding A matrix, B matrix, C matrix, D matrix, and Δ matrix are generated by mapping. The expressions of each matrix are:

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] in, The coordinates are A matrix of pixels, The coordinates are The B matrix of the pixel points, The coordinates are The C matrix of pixels, The coordinates are The D matrix of pixels, The coordinates are The Δ matrix of pixels, The coordinates are The fusion embedding features of the pixel points, 、 、 、 、 are mapping functions, Indicates the state space dimension. In this embodiment, N=8. In this embodiment, the state space dimension is 8. Indicates the number of fusion embedding feature channels. In this embodiment, d=64. Represents the operation of constructing a diagonal matrix.

[0100] Discretize the A matrix, B matrix, C matrix, and D matrix to obtain the corresponding discretization matrices. Specifically, the expressions of the discretization matrices of the A matrix, B matrix, C matrix, and D matrix are:

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] in, 、 、 、 They are matrix, matrix, matrix, The discretization matrix of the matrix, Represents the identity matrix.

[0106] The expressions of the discretization matrices of each pixel in the fusion embedded features are:

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] in, The coordinates are Pixels The discretization matrix of the matrix, The coordinates are Pixels The discretization matrix of the matrix, The coordinates are Pixels The discretization matrix of the matrix, The coordinates are Pixels The discretization matrix of the matrix.

[0112] Calculate the cosine distance between each pixel in the fused embedded feature and the pixels in the four directions of up, down, left, and right with it as the midpoint. The expression is:

[0113] ;

[0114] ;

[0115] ;

[0116] ;

[0117] in, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixels.

[0118] The cosine distance between two pixels is used as the weight value of its weighted edge to obtain the feature construction graph. The expression of the feature construction graph is: ,in, Represents the feature structure diagram, Represents the pixel set of the feature construction map, A weighted edge set representing a feature construction graph.

[0119] The Brufuka algorithm is used to prune the feature construction graph to obtain the minimum spanning tree.

[0120] Combine Figure 4 Finally, the tree scanning module is used to aggregate the information of all pixels in the fusion embedded features using the minimum spanning tree to obtain the spectral fusion state space vector, and fuse the spectral fusion state space vector with the fusion embedded features to obtain the super-resolution fusion reconstructed image. The data processing process specifically includes:

[0121] like Figure 5 As shown in Figure 1, the tree scanning algorithm is used to perform preliminary aggregation of the state space vectors of the pixels in the fused embedded features to obtain the updated state space vectors of the pixels:

[0122] The pixel point is used as the root node of the minimum spanning tree, that is:

[0123] ;

[0124] in, The coordinates are The fusion embedding features of the pixel points, The coordinates are The pixel point is regarded as the root node.

[0125] The root node can determine multiple leaf nodes of the tree, namely:

[0126] ;

[0127] in, Indicates the leaf nodes, Indicates the number of leaf nodes, Represents the root node, represents the minimum spanning tree, According to the root node and the minimum spanning tree Determine the operation of the leaf node.

[0128] According to the properties of the minimum spanning tree, any two points can determine a unique path, so the root node can determine a unique path with each leaf node. Since the path is directionless, all paths are defined from the leaf node to the root node.

[0129] A matrix, B matrix, and Δ matrix are used to perform selective scanning aggregation on the path sequence, and multiple path information is aggregated into the state space vector of the root node.

[0130] The basic formula for selective scanning information aggregation in the Mamba model is as follows:

[0131] ;

[0132] ;

[0133] Among them, i represents the index sequence of any element in the sequence, and the sequence consists of multiple elements. , S represents the sequence length, represents the state space vector of element i, represents the input data of element i, Represents the output data of element i, 、 、 、 Represent the discretization matrices of A matrix, B matrix, C matrix and D matrix of element i respectively, where 、 、 、 、 、 、 , where N is 8 and d is 64. The recursive iteration of state transfer can be performed, taking the sequence length S as 4 as an example, as shown in the formula:

[0134] ;

[0135] General Orders ,but , then:

[0136] ;

[0137] Similarly, we can get:

[0138] ;

[0139] ;

[0140] ;

[0141] By mathematical induction, we can get the information aggregation of the state space vector of the terminal element of a sequence of arbitrary length S during the scanning process, as shown in the formula:

[0142] ;

[0143] Where S is the sequence length, is the state space vector of the element at the end (Sth), 、 is the parameter matrix of each corresponding subscript element in the sequence, The input data for each corresponding subscript element in the sequence.

[0144] If the meaning of the formula is extended so that it not only aggregates the state space vectors of the end elements of the sequence, but also represents the information aggregation of the state space vectors of any element in the sequence, the formula can be extended to:

[0145] ;

[0146] This formula is called the state aggregation formula of path scanning, where: is the index of any element in the sequence, satisfying ,when , indicating that the current element is at the beginning of the sequence, which is the beginning of the scanning operation. , indicating that the current element is at the end of the sequence, which is the end of the scan operation.

[0147] Therefore, for all paths from leaf nodes to the root node R, the state aggregation formula of path scanning can be used to aggregate the information of the root node to obtain the state space vector after the pixel point is updated. The formula is as follows:

[0148] ;

[0149] in, Represents the updated state space vector obtained by taking the pixel point as the root node R, Indicates that Leaf nodes Directed path to the root node R, Indicates that the leaf node Nodes on the directed path to the root node R, Represented by the node Directed path to the root node R, Represented by the node Nodes on the directed path to the root node R, Representation node The A matrix corresponding to the pixel point is discretized matrix, Representation node The B matrix discretization matrix corresponding to the pixel point, Representation node The fused embedding features of the corresponding pixels, Indicates the number of leaf nodes.

[0150] like Figure 6 As shown, the updated state space vector of the pixel point is then subjected to multiple one-dimensional dilated convolutions to obtain multiple corresponding spectral fusion feature outputs. In this embodiment, the convolution kernel size of the dilated convolution is 9, the number of convolution kernels is 64, the dilation rate is 3, and the number of dilated convolutions is 4; and the spectral fusion state space vector of each pixel point is obtained by weighted summation of the multiple corresponding outputs. The formula is as follows:

[0151] ;

[0152] ;

[0153] ;

[0154] Where N represents the number of one-dimensional dilated convolutions, represents the nth one-dimensional dilated convolution operation, represents the dilation rate of the n-th one-dimensional dilated convolution, Represents the spectral fusion features output by n one-dimensional dilated convolutions, The coordinates are The updated state space vector obtained by taking the pixel point as the root node is The coordinates are The spectral fusion state space vector of the pixel point is, The weight of the spectral fusion feature output by the n-th one-dimensional dilated convolution is set as a learnable parameter in this embodiment.

[0155] The discretization matrix of C matrix and D matrix is ​​used to process the spectral fusion state space vector and fusion embedding features of the pixel point to obtain the output feature of the pixel point, which is expressed as follows:

[0156] ;

[0157] ;

[0158] in, The coordinates are The output features of the pixel points, The coordinates are The C matrix of the pixel points is discretized into a matrix, The coordinates are The spectral fusion state space vector of the pixel point is, The coordinates are The D matrix of the pixel points is discretized into a matrix, The coordinates are The fusion embedding features of the pixel points, represents the mapping function, represents the convolution operation, Indicates intermediate operation parameters.

[0159] Finally, the output features of all pixels in the combined fusion embedding feature are Get the super-resolution fusion reconstructed image.

[0160] In this embodiment, the CAVE dataset is used. The CAVE dataset contains 32 sets of raw hyperspectral image data of indoor scenes, of which 20 sets of indoor scenes are used as training sets and 12 sets of indoor scenes are used as test sets. The benchmark task is constructed with an 8-fold spatial scaling ratio.

[0161] This dataset is created by spatially downsampling the original hyperspectral image data (512×512 pixels, 31 bands) (scaling ratio 1 / 8). Specifically, the spatial downsampling involves evenly cropping the original hyperspectral image data into 64 parts, generating a 64×64 low-resolution hyperspectral image (LR-HSI). At the same time, four multispectral bands (RGB + near-infrared) are extracted from the original hyperspectral image data as a high-resolution multispectral image (HR-MSI).

[0162] Taking low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI) as input, the model is required to reconstruct a 512×512 high-resolution hyperspectral image (HR-HSI).

[0163] This partitioning ensures both training data diversity (covering 20 material types, including fabrics and paints) and the generalizability of the test set (12 unseen scenes). Furthermore, the 8x super-resolution ratio fully demonstrates the algorithm's spectral-spatial co-reconstruction capabilities under extreme resolution differences, for example recovering texture details (such as paint strokes) and fine spectral features (such as metallic reflection peaks) from 64×64 LR-HSI. This setup has become a standard protocol for evaluating the robustness of fusion models in metrics such as SAM (target <3°) and PSNR (>42dB).

[0164] The spectral state fusion tree Mamba network model (SSFTM) in this embodiment is compared with a convolutional neural network model (CNN), a residual network model (ResNet), a deep learning model (Transformer), and a multi-scale Inception Mixer Transformer network (MIMFormer). In the task of hyperspectral and multispectral image fusion super-resolution reconstruction, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are typically used to evaluate the ability to preserve spatial detail (larger PSNR and SSIM values ​​are better). PSNR quantifies pixel-level global error, while SSIM measures structural similarity. Spectral angle mapping (SAM) and relative global error (ERGAS) are used to evaluate spectral fidelity (smaller SAM and ERGAS values ​​are better). SAM calculates the average angle between spectral vectors, while ERGAS integrates spatial and spectral errors. Root mean square error (RMSE) directly reflects the pixel-level deviation of the reconstruction result (smaller RMSE values ​​are better). These indicators jointly measure the comprehensive performance of the model in terms of spatial clarity, spectral consistency and overall accuracy. The comparative test results are shown in Table 1:

[0165] Table 1: Comparison of super-resolution reconstruction results of hyperspectral and multispectral images fused by various models on the CAVE dataset

[0166]

[0167] As shown in Table 1, the Spectral State Fusion Tree Mamba Network Model (SSFTM) of this embodiment achieves optimal performance across all metrics: peak signal-to-noise ratio (PSNR) = 44.68 and structural similarity (SSIM) = 0.9925, while spectral angle mapping (SAM) = 1.79°, relative global error (ERGAS) = 1.36, and root mean square error (RMSE) = 0.0057 are the lowest, significantly leading in overall performance. The ResNet model ranks second with a PSNR of 43.42 and structural similarity (SSIM) of 0.9903, but its training time is as long as 700.57 seconds. The training time of the Spectral State Fusion Tree Mamba Network Model (SSFTM) of this embodiment is 668.04 seconds, indicating that the ResNet model has lower training efficiency than the Spectral State Fusion Tree Mamba Network Model (SSFTM) of this embodiment. The convolutional neural network (CNN) model trained the fastest, taking 146.4 seconds, but its peak signal-to-noise ratio (PSNR) was 41.39 and its spectral index (SAM) was 2.51°, which was relatively weak. The deep learning model (Transformer) achieved the lowest PSNR (40.45) due to computational redundancy and took a long time to train (573.99 seconds). While the multi-scale inception mixer transformer network (MIMFormer) outperformed the deep learning model (Transformer) with a PSNR (42.79), it took the longest to train (1135.57 seconds), limiting its practicality. The spectral state fusion tree Mamba network model (SSFTM) of this embodiment achieved the best balance between accuracy and efficiency.

[0168] To visualize the classification results, Figure 7 The paper shows a schematic diagram comparing the super-resolution fusion reconstruction image of the test data "oil painting" on the CAVE dataset using the hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba and the output images of other models. It can be seen that SSFTM has the highest image clarity in the super-resolution reconstruction of hyperspectral and multispectral image fusion. Figure 8 The paper shows a schematic diagram comparing the super-resolution fusion reconstructed images of the test data "real and fake peppers" on the CAVE dataset using the hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba with the output images of other models. It can be seen that the image super-resolution reconstructed by SSFTM in the fusion of hyperspectral and multispectral images has the highest clarity and quality. Figure 9The paper shows a schematic diagram comparing the super-resolution fusion reconstruction image of the test data "plush toys" on the CAVE dataset based on the hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba and the output images of other models. It can be seen that the image clarity of the super-resolution reconstruction of the fusion of hyperspectral and multispectral images by SSFTM is the highest, the difference with the label is the smallest, and it has the best effect.

[0169] Example 2:

[0170] This embodiment provides a hyperspectral and multispectral image fusion system based on a spectral state fusion tree Mamba, including:

[0171] An image acquisition module configured to acquire hyperspectral images and multispectral images;

[0172] The hyperspectral and multispectral image fusion module is configured to take the hyperspectral image and the multispectral image as input, and output a super-resolution fused reconstructed image based on the spectral state fusion tree Mamba network model as described in Example 1.

[0173] Example 3:

[0174] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba as described in Example 1 is implemented.

[0175] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A hyperspectral and multispectral image fusion method based on spectral state fusion tree Mamba, characterized in that: include: Acquire hyperspectral and multispectral images; Taking hyperspectral images and multispectral images as input, the Mamba network model based on spectral state fusion tree is used to spatially upsample the hyperspectral images to obtain high spatial resolution hyperspectral images. The high spatial resolution hyperspectral image and the multispectral image are spliced ​​along the channel dimension to obtain a fused image; Perform convolution operation on the fused image to obtain the fused embedded features; Generate a minimum spanning tree based on the fused embedded features; The minimum spanning tree is used to aggregate information of all pixels in the fusion embedded features to obtain a spectral fusion state space vector, and the spectral fusion state space vector is fused with the fusion embedded features to obtain a super-resolution fusion reconstructed image, including: For each pixel in the fused embedding feature: The tree scanning algorithm is used to perform preliminary aggregation of the state space vectors of the pixels in the fused embedded features to obtain the updated state space vectors of the pixels, including: The pixel point is used as the root node of the minimum spanning tree to determine multiple leaf nodes, and multiple directed paths from the leaf nodes to the root node are obtained based on the root node and the leaf nodes; The discretized matrices of matrix A and matrix B are used to selectively scan and aggregate each directed path to obtain the corresponding directed path information, and all directed path information is aggregated to the state space vector of the root node to obtain the updated state space vector of the pixel point, which is expressed as follows: ； in, Represents the updated state space vector obtained by taking the pixel point as the root node R, Indicates that Leaf nodes Directed path to the root node R, Indicates that the leaf node Nodes on the directed path to the root node R, Represented by the node Directed path to the root node R, Represented by the node Nodes on the directed path to the root node R, Representation node The A matrix corresponding to the pixel point is discretized matrix, Representation node The B matrix discretization matrix corresponding to the pixel point, Representation node The fused embedding features of the corresponding pixels, Indicates the number of leaf nodes; Perform multiple one-dimensional dilated convolutions on the updated state space vector of the pixel to obtain multiple corresponding outputs, and then perform weighted summation of the multiple corresponding outputs to obtain the spectral fusion state space vector of each pixel. The discretized matrix of C matrix and D matrix is ​​used to process the spectral fusion state space vector and fusion embedding feature of the pixel point to obtain the output feature of the pixel point; The output features of all pixels in the combined fusion embedded features are used to obtain a super-resolution fused reconstructed image; The spectral state fusion tree Mamba network model includes a convolution module, a tree construction module and a tree scanning module connected in sequence, and the input end of the convolution module is connected to an upsampling module.

2. The hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba according to claim 1 is characterized in that: The expression of the hyperspectral image is: , the expression of the high spatial resolution hyperspectral image is , the expression of the multispectral image is , the expression of the fused image is , the expression of the fusion embedding feature is ; in, represents the original hyperspectral image, represents a hyperspectral image with high spatial resolution, represents a multispectral image, represents the fused image, Indicates the image height and width in low spatial resolution state, Indicates the image height and width in high spatial resolution state, Indicates the number of channels in low spectral resolution state, Indicates the number of channels in high spectral resolution state, represents the fusion embedding feature, Indicates the number of fused embedding feature channels.

3. The hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba according to claim 1 is characterized in that: Generating a minimum spanning tree according to the fused embedded features includes: For each pixel point in the fusion embedded feature, the corresponding A matrix, B matrix, C matrix, D matrix, and Δ matrix are generated by mapping, and the A matrix, B matrix, C matrix, and D matrix are discretized to obtain the corresponding discretization matrix; Calculate the cosine distance between each pixel point in the fused embedded feature and the pixels in the four directions of up, down, left, and right with it as the midpoint; The cosine distance between two pixels is used as the weight value of its weighted edge to obtain the feature construction graph; The Borufka algorithm is used to prune the feature construction graph to obtain the minimum spanning tree.

4. The hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba according to claim 3 is characterized in that: The expressions of the A matrix, B matrix, C matrix, D matrix and Δ matrix are respectively: ； ； ； ； ； in, The coordinates are A matrix of pixels, The coordinates are The B matrix of the pixel points, The coordinates are The C matrix of pixels, The coordinates are The D matrix of pixels, The coordinates are The Δ matrix of pixel points, The coordinates are The fusion embedding features of the pixel points, 、 、 、 、 are mapping functions, represents the state space dimension, represents the number of fusion embedding feature channels, Indicates the operation of constructing a diagonal matrix; The discretization matrix expressions of the A matrix, B matrix, C matrix, and D matrix are respectively: ； ； ； ； in, 、 、 、 They are matrix, matrix, matrix, The discretization matrix of the matrix, represents the identity matrix; The expressions of the discretization matrices of each pixel in the fused embedded features are: ； ； ； ； in, The coordinates are Pixels The discretization matrix of the matrix, The coordinates are Pixels The discretization matrix of the matrix, The coordinates are Pixels The discretization matrix of the matrix, The coordinates are Pixels The discretization matrix of the matrix.

5. The hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba according to claim 3 is characterized in that: The expression for calculating the cosine distance between each pixel point in the fusion embedded feature and the pixel points in the four directions of the upper, lower, left, and right with the pixel point as the midpoint is: ； ； ； ； in, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points; The expression of the characteristic structure graph is: ,in, Represents the feature structure diagram, Represents the pixel set of the feature construction map, A weighted edge set representing a feature construction graph.

6. The hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba according to claim 1 is characterized in that: The expression of the output feature of the pixel point obtained by processing the spectral fusion state space vector and the fusion embedded feature of the pixel point using the discretization matrix of the C matrix and the D matrix is: ； ； in, The coordinates are The output features of the pixel points, The coordinates are The C matrix of the pixel points is discretized into a matrix, The coordinates are The spectral fusion state space vector of the pixel point is, The coordinates are The D matrix of the pixel points is discretized into a matrix, The coordinates are The fusion embedding features of the pixel points, represents the mapping function, represents the convolution operation, Indicates intermediate operation parameters.

7. A hyperspectral and multispectral image fusion system based on a spectral state fusion tree Mamba, implemented using the hyperspectral and multispectral image fusion method based on a spectral state fusion tree Mamba according to any one of claims 1 to 6, characterized in that: include: An image acquisition module configured to acquire hyperspectral images and multispectral images; The hyperspectral and multispectral image fusion module is configured to take the hyperspectral image and the multispectral image as input, and obtain the super-resolution fused reconstructed image based on the output of the spectral state fusion tree Mamba network model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba is implemented.

Images