Hyperspectral multispectral image fusion method and system based on spectral state fusion tree Mamba and storage medium

Abstract

The invention discloses a hyperspectral and multispectral image fusion method and system based on a spectral state fusion tree Mamba and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: obtaining a hyperspectral image and a multispectral image; the hyperspectral image and the multispectral image serve as input, and a super-resolution fusion reconstruction image is obtained based on output of a spectral state fusion tree Mama network model; the spectrum state fusion tree Mama network model comprises a convolution module, a tree construction module and a tree scanning module which are connected in sequence, and the input end of the convolution module is connected with an up-sampling module. According to the method, a tree Mama scanning mechanism for adaptively scanning a spatial mechanism of hyperspectral image data is utilized, so that efficient hyperspectral spatial information interaction is realized, and the problem that Mama model channels are mutually independent and lack of information interaction is solved; the classification precision of the generated super-resolution fusion reconstruction image is superior to that of an existing image fusion algorithm while the low time complexity is kept.

Description

Technical Field

[0001] The present invention relates to a hyperspectral and multispectral image fusion method, system and storage medium based on a spectral state fusion tree Mamba, belonging to the field of computer vision technology. Background Technique

[0002] Hyperspectral and multispectral image fusion super-resolution reconstruction technology aims to combine the nanoscale spectral resolution of hyperspectral images (HSIs) with the sub-meter spatial resolution of multispectral images (MSIs) to generate fused images with both wide-band coverage and high spatial details, so as to meet the requirements of fine recognition and quantitative analysis of ground objects in scenarios such as satellite remote sensing and environmental monitoring.

[0003] However, hyperspectral data is usually limited by sensor hardware and has a low spatial resolution (such as 30 meters / pixel), while multispectral data, although having a higher spatial resolution (1-5 meters / pixel), has a limited number of spectral channels (only 4-8 bands). There is a natural spectral-spatial resolution inversion contradiction between the two.

[0004] Although methods based on convolutional neural networks (CNNs) can extract spatial texture features through local convolutional kernels, their limited receptive fields are difficult to capture long-range spectral correlations across hundreds of bands. Although Transformer-based models use self-attention mechanisms to enhance global modeling capabilities, their quadratic computational complexity leads to a sharp increase in memory overhead when processing large-scale remote sensing images. The emerging Mamba architecture achieves global modeling with linear complexity through a state space model (SSM), but its fixed row-column scanning path cannot adapt to the irregular spatial distribution characteristics of ground objects in hyperspectral images, and the channel-independent processing mechanism severs the interaction of cross-band spectral features.

[0005] Therefore, existing methods are difficult to achieve an effective balance among computational efficiency, spectral fidelity, and spatial detail preservation, which has become the core obstacle restricting the practical application of hyperspectral and multispectral fusion technology. Summary of the Invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a hyperspectral and multispectral image fusion method, system and storage medium based on a spectral state fusion tree Mamba. By using the tree Mamba scanning mechanism that adaptively scans the spatial structure of hyperspectral image data, efficient hyperspectral spatial information interaction is achieved. By spectral state fusion, the problem that the Mamba model channels are independent of each other and lack information interaction is overcome. The generated super-resolution fusion reconstruction image has a classification accuracy superior to that of existing image fusion algorithms while maintaining a low time complexity.

[0007] To achieve the above object, the present invention is implemented by the following technical solutions:

[0008] In a first aspect, the present invention provides a hyperspectral and multispectral image fusion method based on a spectral state fusion tree Mamba, comprising:

[0009] Acquire hyperspectral and multispectral images;

[0010] Taking hyperspectral images and multispectral images as input, the Mamba network model based on spectral state fusion tree is:

[0011] Perform spatial upsampling on the hyperspectral image to obtain a hyperspectral image with high spatial resolution;

[0012] The high spatial resolution hyperspectral image and the multispectral image are spliced along the channel dimension to obtain a fused image;

[0013] Perform convolution operation on the fused image to obtain the fused embedded features;

[0014] Generate a minimum spanning tree based on the fused embedded features;

[0015] The minimum spanning tree is used to aggregate information of all pixels in the fused embedded features to obtain the spectral fusion state space vector, and the spectral fusion state space vector is fused with the fused embedded features to obtain the super-resolution fused reconstructed image;

[0016] The spectral state fusion tree Mamba network model includes a convolution module, a tree construction module and a tree scanning module connected in sequence, and the input end of the convolution module is connected to an upsampling module.

[0017] Furthermore, the expression of the hyperspectral image is: , the expression of the high spatial resolution hyperspectral image is , the expression of the multispectral image is , the expression of the fused image is , the expression of the fusion embedding feature is ;

[0018] in, represents the original hyperspectral image, represents a hyperspectral image with high spatial resolution, represents a multispectral image, represents the fused image, Indicates the image height and width in low spatial resolution state, Indicates the image height and width in high spatial resolution state, Indicates the number of channels in low spectral resolution state, Indicates the number of channels in high spectral resolution state, represents the fusion embedding feature, Indicates the number of fused embedding feature channels.

[0019] Furthermore, the data processing process of generating a minimum spanning tree based on the fused embedded features specifically includes:

[0020] For each pixel point in the fusion embedded feature, the corresponding A matrix, B matrix, C matrix, D matrix, and Δ matrix are generated by mapping, and the A matrix, B matrix, C matrix, and D matrix are discretized to obtain the corresponding discretization matrix;

[0021] Calculate the cosine distance between each pixel point in the fused embedded feature and the pixels in the four directions of up, down, left, and right with it as the midpoint;

[0022] The cosine distance between two pixels is used as the weight value of its weighted edge to obtain the feature construction graph;

[0023] The Borufka algorithm is used to prune the feature construction graph to obtain the minimum spanning tree.

[0024] Furthermore, the expressions of the A matrix, B matrix, C matrix, D matrix, and Δ matrix are respectively:

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] in, The coordinates are A matrix of pixels, The coordinates are The B matrix of the pixel points, The coordinates are The C matrix of pixels, The coordinates are The D matrix of pixels, The coordinates are The Δ matrix of pixels, The coordinates are The fusion embedding features of the pixel points, 、 、 、 、 are mapping functions, represents the state space dimension, represents the number of fusion embedding feature channels, Indicates the operation of constructing a diagonal matrix;

[0031] The discretization matrix expressions of the A matrix, B matrix, C matrix, and D matrix are respectively:

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] in, 、 、 、 They are matrix, matrix, matrix, The discretization matrix of the matrix, represents the identity matrix;

[0037] The expressions of the discretization matrices of each pixel in the fused embedded features are:

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] in, The coordinates are Pixels The discretization matrix of the matrix, The coordinates are Pixels The discretization matrix of the matrix, The coordinates are Pixels The discretization matrix of the matrix, The coordinates are Pixels The discretization matrix of the matrix.

[0043] Furthermore, the expression for calculating the cosine distance between each pixel point in the fusion embedded feature and the pixel points in the four directions of up, down, left, and right with the pixel point as the midpoint is:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] in, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The pixel points and coordinates are The cosine distance between pixels, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points;

[0049] The expression of the characteristic structure graph is: ,in, Represents the feature structure diagram, Represents the pixel set of the feature construction map, A weighted edge set representing a feature construction graph.

[0050] Furthermore, the method of using the minimum spanning tree to aggregate information of all pixels in the fused embedded features to obtain a spectral fusion state space vector, and fusing the spectral fusion state space vector with the fused embedded features to obtain a super-resolution fused reconstructed image includes:

[0051] For each pixel in the fused embedding feature:

[0052] The tree scanning algorithm is used to perform preliminary aggregation of the state space vector of the pixel points in the fused embedded features to obtain the updated state space vector of the pixel points;

[0053] Perform multiple one-dimensional dilated convolutions on the updated state space vector of the pixel to obtain multiple corresponding outputs, and then perform weighted summation of the multiple corresponding outputs to obtain the spectral fusion state space vector of each pixel.

[0054] The discretized matrix of C matrix and D matrix is used to process the spectral fusion state space vector and fusion embedding feature of the pixel point to obtain the output feature of the pixel point;

[0055] The output features of all pixels in the embedded features are combined to obtain a super-resolution fused reconstructed image.

[0056] Furthermore, the method of performing preliminary aggregation of the state space vectors of the pixels using the tree scanning algorithm to obtain the updated state space vectors of the pixels includes:

[0057] The pixel point is used as the root node of the minimum spanning tree to determine multiple leaf nodes, and multiple directed paths from the leaf nodes to the root node are obtained based on the root node and the leaf nodes;

[0058] The discretized matrices of matrix A and matrix B are used to selectively scan and aggregate each directed path to obtain the corresponding directed path information, and all directed path information is aggregated to the state space vector of the root node to obtain the updated state space vector of the pixel point, which is expressed as follows:

[0059] ;

[0060] in, Represents the updated state space vector obtained by taking the pixel point as the root node R, Indicates that Leaf nodes Directed path to the root node R, Indicates that the leaf node Nodes on the directed path to the root node R, Represented by the node Directed path to the root node R, Represented by the node Nodes on the directed path to the root node R, Representation Node The A matrix corresponding to the pixel point is discretized matrix, Representation Node The B matrix corresponding to the pixel point is discretized matrix, Representation Node The fused embedding features of the corresponding pixels, Indicates the number of leaf nodes.

[0061] Furthermore, the expression for obtaining the output feature of a pixel point by processing the spectral fusion state space vector and the fusion embedding feature of the pixel point using the discretization matrices of the C matrix and the D matrix is:

[0062] ;

[0063] ;

[0064] Among them, represents the output feature of the pixel point with coordinates , represents the C matrix discretization matrix of the pixel point with coordinates , represents the spectral fusion state space vector of the pixel point with coordinates , represents the D matrix discretization matrix of the pixel point with coordinates , represents the fusion embedding feature of the pixel point with coordinates , represents the mapping function, represents the convolution operation, represents the intermediate operation parameter.

[0065] In a second aspect, the present invention further provides a hyperspectral and multispectral image fusion system based on a spectral state fusion tree Mamba, including:

[0066] An image acquisition module configured to acquire a hyperspectral image and a multispectral image;

[0067] A hyperspectral and multispectral image fusion module configured to take the hyperspectral image and the multispectral image as inputs and output a super-resolution fusion reconstruction image based on the spectral state fusion tree Mamba network model.

[0068] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it implements the hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba as described in any item of the first aspect.

[0069] Compared with the prior art, the beneficial effects achieved by the present invention:

[0070] The present invention utilizes the tree Mamba scanning mechanism for spatially adaptive scanning of hyperspectral image data, achieving efficient hyperspectral spatial information interaction. The state space vector obtained by aggregating tree scanning information is used to implement spectral state fusion through one-dimensional dilated convolution on the channel dimension, overcoming the problem that the channels of the Mamba model are independent of each other and lack information interaction. Finally, the obtained spectral fusion state space vector is fused with the input fusion embedding feature to generate a super-resolution fusion reconstruction image, which has a classification accuracy superior to existing image fusion algorithms while maintaining a low time complexity.

[0071] The present invention proposes a scanning method for tree Mamba, which scans the two-dimensional image in the order of the minimum spanning tree structure. It can arrange the pixel points with high spatial similarity and spectral similarity in relatively close positions in the scanning sequence, and arrange the pixel points with low spatial similarity and spectral similarity in relatively far positions in the scanning sequence, reasonably constructing the distance relationship of pixel points in the scanning sequence. At the same time, the minimum spanning tree can adapt to the spatial structure distribution of pixel points, which is beneficial to the modeling of the Mamba model on two-dimensional image data, and performs one-dimensional convolution on the intermediate state space vector of the scan along the spectral dimension to achieve spectral information fusion, further making full use of the rich physical information in the spectral dimension. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a schematic flow chart of a hyperspectral and multispectral image fusion method based on spectral state fusion tree Mamba in an embodiment of the present invention;

[0073] Figure 2 It is a schematic structural diagram of a spectral state fusion tree Mamba network model in an embodiment of the present invention;

[0074] Figure 3 It is a schematic diagram of the data processing process of the tree construction module of the spectral state fusion tree Mamba network model in an embodiment of the present invention;

[0075] Figure 4 It is a schematic diagram of the data processing process of the tree scanning module of the spectral state fusion tree Mamba network model in an embodiment of the present invention;

[0076] Figure 5 It is a schematic diagram of the preliminary aggregation of the state space vector in the tree scanning module of the spectral state fusion tree Mamba network model in an embodiment of the present invention;

[0077] Figure 6 It is a schematic diagram of the one-dimensional dilated convolution processing in the tree scanning module of the spectral state fusion tree Mamba network model in an embodiment of the present invention;

[0078] Figure 7Schematic diagram showing a comparison between a super-resolution fused reconstructed image of a test data "oil painting" using a hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba and output images from other models on a CAVE dataset in one embodiment of the present invention;

[0079] Figure 8 Schematic diagram showing a comparison between a super-resolution fused reconstructed image of the test data "real and fake peppers" and output images of other models using a hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba on a CAVE dataset in one embodiment of the present invention;

[0080] Figure 9 This is a schematic diagram showing a comparison between a super-resolution fused reconstructed image of the test data "plush toys" on the CAVE dataset using a hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba in an embodiment of the present invention and images output by other models. DETAILED DESCRIPTION

[0081] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0082] Embodiment 1:

[0083] The embodiment of the present invention provides a hyperspectral and multispectral image fusion method based on a spectral state fusion tree Mamba, comprising the following steps:

[0084] First, hyperspectral images and multispectral images are acquired.

[0085] The expression of hyperspectral image is: , the expression of multispectral image is ,in, represents the original hyperspectral image, represents a multispectral image, Indicates the image height and width in high spatial resolution state, Indicates the number of channels in low spectral resolution state, Indicates the number of channels in high spectral resolution state.

[0086] Taking hyperspectral images and multispectral images as input, a high-resolution fused reconstructed image is output based on the spectral state fusion tree Mamba network model.

[0087] The structure of the spectral state fusion tree Mamba network model is as follows Figure 2As shown, it includes a convolution module, a tree construction module and a tree scanning module connected in sequence, and the input end of the convolution module is also connected to the output end of the upsampling module.

[0088] like Figure 1 As shown, the hyperspectral image is input from the input end of the upsampling module, and the upsampling module spatially upsamples the hyperspectral image to obtain a hyperspectral image with high spatial resolution, which is expressed as , represents a hyperspectral image with high spatial resolution, Indicates the image height and width in high spatial resolution state, Indicates the number of channels in high spectral resolution state.

[0089] The high spatial resolution hyperspectral image and multispectral image are spliced along the channel dimension to obtain a fused image. The fused image is input into the convolution module for convolution operation to obtain the fused embedded feature. In this embodiment, the convolution kernel size is 3 and the number of convolution kernels is 64. The specific operation can be expressed as follows:

[0090] ;

[0091] in, represents the fusion embedding feature, , ,in, The coordinates are The fusion embedding features of the pixel points, Indicates the image height and width in high spatial resolution state, represents the number of fusion embedding feature channels, represents the convolution operation, represents the fused image, .

[0092] Combine Figure 3 ,The tree construction module is used to generate a minimum spanning tree based on the fused embedding features, specifically including:

[0093] For each pixel in the fusion embedded feature, the corresponding A matrix, B matrix, C matrix, D matrix, and Δ matrix are generated by mapping. The expressions of each matrix are:

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] Among them, represents the A matrix of the pixel point with coordinates , represents the B matrix of the pixel point with coordinates , represents the C matrix of the pixel point with coordinates , represents the D matrix of the pixel point with coordinates , represents the Δ matrix of the pixel point with coordinates , represents the fused embedding feature of the pixel point with coordinates , , , , , are all mapping functions, represents the state space dimension. In this embodiment, N = 8. In this embodiment, the state space dimension is 8, represents the number of channels of the fused embedding feature. In this embodiment, d = 64, represents the operation of constructing a diagonal matrix.

[0100] The A matrix, B matrix, C matrix, and D matrix are discretized to obtain the corresponding discretized matrices. Specifically, the expressions of the discretized matrices of the A matrix, B matrix, C matrix, and D matrix are respectively:

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] Among them, , , , are respectively matrix, matrix, matrix, the discretized matrices of the matrix, represents the identity matrix.

[0106] The expressions of the respective discretized matrices of each pixel point in the fused embedding feature are:

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] Among them, represents the discretized matrix of the matrix of the pixel point with coordinates ; represents the discretized matrix of the matrix of the pixel point with coordinates ; represents the discretized matrix of the matrix of the pixel point with coordinates ; represents the discretized matrix of the matrix of the pixel point with coordinates ;

[0112] Calculate the cosine distance between each pixel point in the fused embedded feature and the pixel points in the four directions of up, down, left, and right with it as the midpoint. Its expression is:

[0113] ;

[0114] ;

[0115] ;

[0116] ;

[0117] Among them, represents the cosine distance between the pixel point with coordinates and the pixel point with coordinates ; represents the cosine distance between the pixel point with coordinates and the pixel point with coordinates ; represents the cosine distance between the pixel point with coordinates and the pixel point with coordinates ; represents the cosine distance between the pixel point with coordinates and the pixel point with coordinates ; represents the fused embedded feature of the pixel point with coordinates ; represents the coordinates The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixel points, The coordinates are The fusion embedding features of the pixels.

[0118] The cosine distance between two pixels is used as the weight value of its weighted edge to obtain the feature construction graph. The expression of the feature construction graph is: ,in, Represents the feature structure diagram, Represents the pixel set of the feature construction map, A weighted edge set representing a feature construction graph.

[0119] The Brufuka algorithm is used to prune the feature construction graph to obtain the minimum spanning tree.

[0120] Combine Figure 4 Finally, the tree scanning module is used to aggregate the information of all pixels in the fusion embedded features using the minimum spanning tree to obtain the spectral fusion state space vector, and fuse the spectral fusion state space vector with the fusion embedded features to obtain the super-resolution fusion reconstructed image. The data processing process specifically includes:

[0121] like Figure 5 As shown in Figure 1, the tree scanning algorithm is used to perform preliminary aggregation of the state space vectors of the pixels in the fused embedded features to obtain the updated state space vectors of the pixels:

[0122] The pixel point is used as the root node of the minimum spanning tree, that is:

[0123] ;

[0124] in, The coordinates are The fusion embedding features of the pixel points, The coordinates are The pixel point is regarded as the root node.

[0125] The root node can determine multiple leaf nodes of the tree, namely:

[0126] ;

[0127] in, Indicates the leaf nodes, Indicates the number of leaf nodes, Represents the root node, represents the minimum spanning tree, Indicates an operation to determine leaf nodes based on the root node and the minimum spanning tree Determine the operation of leaf nodes.

[0128] According to the properties of the minimum spanning tree, any two points can determine a unique path. Therefore, the root node can determine a unique path with each leaf node. Since the path is undirected, all paths are defined to point from the leaf node to the root node.

[0129] Perform selective scan aggregation on the path sequence using the A matrix, B matrix, and Δ matrix, and aggregate multiple path information into the state space vector of the root node.

[0130] The basic formula for selective scan information aggregation of the Mamba model is as follows:

[0131] ;

[0132] ;

[0133] where i represents the index sequence of any element in the sequence, the sequence consists of multiple elements, , S represents the sequence length, represents the state space vector of element i, represents the input data of element i, represents the output data of element i, , , , respectively represent the discretized matrices of the A matrix, B matrix, C matrix, and D matrix of element i, where , , , , , , , where N takes 8 and d takes 64, the recursive formula of state transfer can be iterated. Taking the sequence length S as 4 as an example, the formula is:

[0134] ;

[0135] Generally, let , then , then there is:

[0136] ;

[0137] Similarly, it can be obtained:

[0138] ;

[0139] ;

[0140] ;

[0141] It can be obtained by mathematical induction that the information aggregation of the state space vector of the end element during the scanning of any sequence with length S is as shown in the formula:

[0142] ;

[0143] where S is the sequence length, is the state space vector of the element at the end (the S-th), , are the parameter matrices of the corresponding subscript elements in the sequence, is the input data of the corresponding subscript elements in the sequence.

[0144] If the meaning of the formula is generalized so that it not only aggregates the state space vector of the end element of the sequence but represents the information aggregation of the state space vector of any element in the sequence, the formula can be generalized as:

[0145] ;

[0146] This formula is called the state aggregation formula for path scanning. In the formula, is the index of any element in the sequence, satisfying , when , it indicates that the current element is at the beginning of the sequence and is the start of the scanning operation. When , it indicates that the current element is at the end of the sequence and is the end of the scanning operation.

[0147] Therefore, for all paths from leaf nodes to the root node R of the root node, the state aggregation formula for path scanning can be used to aggregate information for the root node, obtaining the updated state space vector of the pixel point. The formula is as follows:

[0148] ;

[0149] where, represents the updated state space vector obtained when the pixel point is used as the root node R, represents the directed path from the -th leaf node to the root node R, represents the nodes on the directed path from the leaf node to the root node R, represents the directed path from the node to the root node R, represents the nodes on the directed path from the node to the root node R, Representation node The A matrix corresponding to the pixel point is discretized matrix, Representation node The B matrix discretization matrix corresponding to the pixel point, Representation node The fused embedding features of the corresponding pixels, Indicates the number of leaf nodes.

[0150] like Figure 6 As shown, the updated state space vector of the pixel point is then subjected to multiple one-dimensional dilated convolutions to obtain multiple corresponding spectral fusion feature outputs. In this embodiment, the convolution kernel size of the dilated convolution is 9, the number of convolution kernels is 64, the dilation rate is 3, and the number of dilated convolutions is 4; and the spectral fusion state space vector of each pixel point is obtained by weighted summation of the multiple corresponding outputs. The formula is as follows:

[0151] ;

[0152] ;

[0153] ;

[0154] Where N represents the number of one-dimensional dilated convolutions, represents the nth one-dimensional dilated convolution operation, represents the dilation rate of the n-th one-dimensional dilated convolution, Represents the spectral fusion features output by n one-dimensional dilated convolutions, The coordinates are The updated state space vector obtained by taking the pixel point as the root node is The coordinates are The spectral fusion state space vector of the pixel point is, The weight of the spectral fusion feature output by the n-th one-dimensional dilated convolution is set as a learnable parameter in this embodiment.

[0155] The discretization matrix of C matrix and D matrix is used to process the spectral fusion state space vector and fusion embedding features of the pixel point to obtain the output feature of the pixel point, which is expressed as follows:

[0156] ;

[0157] ;

[0158] in, The coordinates are The output features of the pixel points, The coordinates are The C matrix of the pixel points is discretized into a matrix, The coordinates are The spectral fusion state space vector of the pixel point is, The coordinates are The D matrix of the pixel points is discretized into a matrix, The coordinates are The fusion embedding features of the pixel points, represents the mapping function, represents the convolution operation, Indicates intermediate operation parameters.

[0159] Finally, the output features of all pixels in the combined fusion embedding feature are Get the super-resolution fusion reconstructed image.

[0160] In this embodiment, the CAVE dataset is used. The CAVE dataset contains 32 sets of raw hyperspectral image data of indoor scenes, of which 20 sets of indoor scenes are used as training sets and 12 sets of indoor scenes are used as test sets. The benchmark task is constructed with an 8-fold spatial scaling ratio.

[0161] This dataset is created by spatially downsampling the original hyperspectral image data (512×512 pixels, 31 bands) (scaling ratio 1 / 8). Specifically, the spatial downsampling involves evenly cropping the original hyperspectral image data into 64 parts, generating a 64×64 low-resolution hyperspectral image (LR-HSI). At the same time, four multispectral bands (RGB + near-infrared) are extracted from the original hyperspectral image data as a high-resolution multispectral image (HR-MSI).

[0162] Taking low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI) as input, the model is required to reconstruct a 512×512 high-resolution hyperspectral image (HR-HSI).

[0163] This partitioning ensures both training data diversity (covering 20 material types, including fabrics and paints) and the generalizability of the test set (12 unseen scenes). Furthermore, the 8x super-resolution ratio fully demonstrates the algorithm's spectral-spatial co-reconstruction capabilities under extreme resolution differences, for example recovering texture details (such as paint strokes) and fine spectral features (such as metallic reflection peaks) from 64×64 LR-HSI. This setup has become a standard protocol for evaluating the robustness of fusion models in metrics such as SAM (target <3°) and PSNR (>42dB).

[0164] The spectral state fusion tree Mamba network model (SSFTM) in this embodiment is compared with a convolutional neural network model (CNN), a residual network model (ResNet), a deep learning model (Transformer), and a multi-scale Inception Mixer Transformer network (MIMFormer). In the task of hyperspectral and multispectral image fusion super-resolution reconstruction, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are typically used to evaluate the ability to preserve spatial detail (larger PSNR and SSIM values are better). PSNR quantifies pixel-level global error, while SSIM measures structural similarity. Spectral angle mapping (SAM) and relative global error (ERGAS) are used to evaluate spectral fidelity (smaller SAM and ERGAS values are better). SAM calculates the average angle between spectral vectors, while ERGAS integrates spatial and spectral errors. Root mean square error (RMSE) directly reflects the pixel-level deviation of the reconstruction result (smaller RMSE values are better). These indicators jointly measure the comprehensive performance of the model in terms of spatial clarity, spectral consistency and overall accuracy. The comparative test results are shown in Table 1:

[0165] Table 1: Comparison of super-resolution reconstruction results of hyperspectral and multispectral images fused by various models on the CAVE dataset

[0166]

[0167] As can be seen from the data in Table 1, the spectral state fusion tree Mamba network model (SSFTM) of this embodiment reaches the optimal level in all indicators: the peak signal-to-noise ratio (PSNR) = 44.68, the structural similarity (SSIM) = 0.9925 is the highest, the spectral angle mapper (SAM) = 1.79°, the relative global error (ERGAS) = 1.36, and the root mean square error (RMSE) = 0.0057 is the lowest, with significantly leading comprehensive performance. The residual network model (ResNet) follows with a peak signal-to-noise ratio (PSNR) = 43.42 and a structural similarity (SSIM) = 0.9903, but the training time is as high as 700.57 seconds. The training time of the spectral state fusion tree Mamba network model (SSFTM) of this embodiment is 668.04 seconds, and the training efficiency of the residual network model (ResNet) is lower than that of the spectral state fusion tree Mamba network model (SSFTM) of this embodiment. The convolutional neural network model (CNN) has a training time = 146.4 seconds and is the fastest to train, but the peak signal-to-noise ratio (PSNR) = 41.39 and the spectral index (SAM) = 2.51° perform weakly; the deep learning model (Transformer) has the lowest peak signal-to-noise ratio (PSNR) of 40.45 and a long training time (training time = 573.99 seconds) due to computational redundancy, while the multi-scale Inception Mixer Transformer network (MIMFormer) has a peak signal-to-noise ratio (PSNR) = 42.79 which is better than that of the deep learning model (Transformer), but the training time is the longest (1135.57 seconds), with limited practicality. The spectral state fusion tree Mamba network model (SSFTM) of this embodiment achieves the best balance between accuracy and efficiency.

[0168] To visualize the classification results, Figure 7 Fig. shows a comparison diagram of the super-resolution fusion reconstruction images of the test data "oil painting" on the CAVE dataset by the hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba and the output images of other models. It can be seen that SSFTM has the highest image clarity in the super-resolution reconstruction of hyperspectral and multispectral image fusion. Figure 8 Fig. shows a comparison diagram of the super-resolution fusion reconstruction images of the test data "real and fake peppers" on the CAVE dataset by the hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba and the output images of other models. It can be seen that SSFTM has the highest image clarity and the highest quality in the super-resolution reconstruction of hyperspectral and multispectral image fusion. Figure 9It shows a comparison diagram of the super-resolution fusion reconstruction image of the hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba for the test data "stuffed toy" on the CAVE dataset with the output images of other models. It can be seen that SSFTM has the highest image clarity in the super-resolution reconstruction of hyperspectral and multispectral image fusion, the smallest difference from the label, and has the best effect.

[0169] Embodiment 2:

[0170] This embodiment provides a hyperspectral and multispectral image fusion system based on the spectral state fusion tree Mamba, including:

[0171] An image acquisition module, configured to acquire a hyperspectral image and a multispectral image;

[0172] A hyperspectral and multispectral image fusion module, configured to take the hyperspectral image and the multispectral image as inputs and output a super-resolution fusion reconstruction image based on the spectral state fusion tree Mamba network model as described in Embodiment 1.

[0173] Embodiment 3:

[0174] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba as described in Embodiment 1.

[0175] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and deformations can be made, and these improvements and deformations should also be regarded as the protection scope of the present invention.

Claims

1. A hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba, characterized in that, Including: Obtain hyperspectral images and multispectral images; Taking the hyperspectral image and the multispectral image as inputs, based on the spectral state fusion tree Mamba network model: upsample the hyperspectral image spatially to obtain a hyperspectral image with high spatial resolution; Concatenate the hyperspectral image with high spatial resolution and the multispectral image along the channel dimension to obtain a fused image; Perform a convolution operation on the fused image to obtain fused embedding features; Generate a minimum spanning tree according to the fused embedding features; Use the minimum spanning tree to aggregate information of all pixel points in the fused embedding features to obtain a spectral fusion state space vector, and fuse the spectral fusion state space vector with the fused embedding features to obtain a super-resolution fused reconstruction image; Wherein, the spectral state fusion tree Mamba network model includes a convolution module, a tree construction module and a tree scanning module connected in sequence, and an upsampling module is connected to the input end of the convolution module.

2. The hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba according to claim 1, wherein The expression of the hyperspectral image is , the expression of the hyperspectral image with high spatial resolution is , the expression of the multispectral image is , the expression of the fused image is , the expression of the fused embedded feature is ; Among them, represents the original hyperspectral image, represents the hyperspectral image with high spatial resolution, represents the multispectral image, represents the fused image, represents the height and width of the image in the low spatial resolution state, represents the height and width of the image in the high spatial resolution state, represents the number of channels in the low spectral resolution state, represents the number of channels in the high spectral resolution state, represents the fused embedded feature, represents the number of channels of the fused embedded feature.

3. The hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba according to claim 1, wherein The generating a minimum spanning tree according to the fused embedding features includes: For each pixel point in the fused embedding features, generate its corresponding A matrix, B matrix, C matrix, D matrix, and Δ matrix through mapping, and discretize the A matrix, B matrix, C matrix, and D matrix to obtain corresponding discretized matrices; Calculate the cosine distance between each pixel point in the fused embedding features and the pixel points in the four directions of up, down, left, and right with it as the midpoint; Take the cosine distance between two pixel points as the weight value of its weighted edge to obtain a feature construction graph; Use the Borůvka algorithm to prune the feature construction graph to obtain a minimum spanning tree.

4. The hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba according to claim 3, wherein The expressions of the A matrix, B matrix, C matrix, D matrix, and Δ matrix are respectively: ； ； ； ； ； Among them, represents the A matrix of the pixel point with coordinates , represents the B matrix of the pixel point with coordinates , represents the C matrix of the pixel point with coordinates , represents the D matrix of the pixel point with coordinates , represents the Δ matrix of the pixel point with coordinates , represents the fused embedding feature of the pixel point with coordinates , , , , , are all mapping functions, represents the state space dimension, represents the number of channels of the fused embedding feature, represents the operation of constructing a diagonal matrix; The expressions of the discretized matrices of the A matrix, B matrix, C matrix, and D matrix are respectively: ； ； ； ； Among them, , , , are respectively matrix, matrix, matrix, the discretized matrix of the matrix, represents the identity matrix; The expressions of the discretized matrices of each pixel point in the fused embedding features are respectively: ； ； ； ； Among them, represents the discretization matrix of the pixel point with coordinates matrix, represents the discretization matrix of the pixel point with coordinates matrix, represents the discretization matrix of the pixel point with coordinates matrix, represents the discretization matrix of the pixel point with coordinates matrix.

5. The hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba according to claim 3, wherein The expression for calculating the cosine distance between each pixel point in the fused embedding features and the pixel points in the four directions of up, down, left, and right with it as the midpoint is: ； ； ； ； Among them, represents the cosine distance between the pixel point with coordinates and the pixel point with coordinates . represents the cosine distance between the pixel point with coordinates and the pixel point with coordinates . represents the cosine distance between the pixel point with coordinates and the pixel point with coordinates . represents the cosine distance between the pixel point with coordinates and the pixel point with coordinates . represents the fused embedding feature of the pixel point with coordinates . represents the fused embedding feature of the pixel point with coordinates . represents the fused embedding feature of the pixel point with coordinates . represents the fused embedding feature of the pixel point with coordinates . represents the fused embedding feature of the pixel point with coordinates ; The expression of the feature structure diagram is , where represents the feature structure diagram,[[]] represents the set of pixel points of the feature structure diagram,[[]] represents the set of weighted edges of the feature structure diagram.[[]] 6. The hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba according to claim 1, wherein The using the minimum spanning tree to aggregate information of all pixel points in the fused embedding features to obtain a spectral fusion state space vector, and fusing the spectral fusion state space vector with the fused embedding features to obtain a super-resolution fused reconstruction image includes: For each pixel point in the fused embedding features: Use the tree scanning algorithm to perform preliminary aggregation of the state space vectors of the pixel points in the fused embedding features to obtain the updated state space vector of the pixel points; Perform multiple one-dimensional dilation convolutions on the updated state space vector of the pixel points to obtain multiple corresponding outputs, and perform weighted summation on the multiple corresponding outputs to obtain the spectral fusion state space vector of each pixel point; Use the discretized matrices of the C matrix and D matrix to process the spectral fusion state space vector and the fused embedding features of the pixel points to obtain the output features of the pixel points; Combine the output features of all pixel points in the fused embedding features to obtain a super-resolution fused reconstruction image.

7. The hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba according to claim 6, wherein The using the tree scanning algorithm to perform preliminary aggregation of the state space vectors of the pixel points to obtain the updated state space vector of the pixel points includes: Take the pixel point as the root node of the minimum spanning tree, thereby determining multiple leaf nodes, and obtaining multiple directed paths from the leaf nodes to the root node according to the root node and the leaf nodes; Use the discretization matrices of matrix A and matrix B to perform selective scan aggregation on each directed path to obtain the corresponding directed path information, and aggregate all the directed path information to the state space vector of the root node to obtain the updated state space vector of the pixel point. Its expression is: ； in, Represents the updated state space vector obtained by taking the pixel point as the root node R, Indicates that Leaf nodes Directed path to the root node R, Indicates that the leaf node Nodes on the directed path to the root node R, Represented by the node Directed path to the root node R, Represented by the node Nodes on the directed path to the root node R, Representation Node The A matrix corresponding to the pixel point is discretized matrix, Representation Node The B matrix corresponding to the pixel point is discretized matrix, Representation Node The fused embedding features of the corresponding pixels, Indicates the number of leaf nodes.

8. The hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba according to claim 6, characterized in that, The expression for obtaining the output feature of the pixel point by processing the spectral fusion state space vector and the fusion embedding feature of the pixel point using the discretization matrices of matrix C and matrix D is: ； ； Among them, represents the output feature of the pixel point with coordinates . represents the C matrix discretization matrix of the pixel point with coordinates . represents the spectral fusion state space vector of the pixel point with coordinates . represents the D matrix discretization matrix of the pixel point with coordinates . represents the fusion embedding feature of the pixel point with coordinates . represents the mapping function, represents the convolution operation, represents the intermediate operation parameter.

9. A hyperspectral and multispectral image fusion system based on the spectral state fusion tree Mamba, characterized in that, Includes: An image acquisition module configured to acquire a hyperspectral image and a multispectral image; A hyperspectral and multispectral image fusion module configured to take the hyperspectral image and the multispectral image as inputs and output a super-resolution fusion reconstruction image based on the spectral state fusion tree Mamba network model.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the hyperspectral and multispectral image fusion method based on the spectral state fusion tree Mamba as described in any one of claims 1 to 8.

Images