A hyperspectral and LiDAR image collaborative recognition method based on Mamba attention fusion

Abstract

This invention relates to hyperspectral and LiDAR remote sensing image fusion and recognition technology. Considering the natural complementarity between hyperspectral images and LiDAR data in terms of spectral features and elevation information, and the problems of existing methods introducing redundant interference through simple feature stitching, this invention provides a collaborative recognition method for hyperspectral and LiDAR images based on Mamba attention fusion. This invention constructs a dual-branch convolutional feature extractor to extract shallow spatial-spectral features of both modalities; introduces a spatial Mamba module for multi-scale global-local long-range dependency feature mining; and designs a dual-attention interactive guided fusion module to achieve deep interaction and fine fusion of cross-modal complementary information through a parallel mechanism of channel attention and spatial attention. This invention effectively suppresses redundant information interference, significantly improves the accuracy and robustness of ground feature fusion classification, and provides high-quality ground feature classification support for remote sensing applications such as urban planning.

Description

Technical Field

[0001] This invention relates to hyperspectral and LiDAR remote sensing image fusion and recognition technology, specifically to a hyperspectral and LiDAR image collaborative recognition method based on Mamba attention fusion. Background Technology

[0002] Joint classification using hyperspectral imagery (HSI) and lidar (LiDAR) has become a research hotspot in remote sensing, as it can simultaneously utilize rich spectral and elevation information. Hyperspectral images cover dozens to hundreds of consecutive bands, providing a unique "spectral fingerprint" for each pixel. However, their spatial resolution is relatively low, and they are susceptible to external factors such as clouds, leading to spectral confusion issues such as "different spectra for the same object" and "different objects with the same spectrum." LiDAR data can accurately acquire three-dimensional spatial information of ground features, generating high-precision digital surface models, compensating for the missing vertical dimension information in hyperspectral images, and effectively alleviating the "different objects with the same spectrum" problem. Therefore, the combined use of hyperspectral and LiDAR data provides an important approach to improving the accuracy of ground feature classification.

[0003] Currently, deep learning methods for hyperspectral image and LiDAR fusion classification tasks mainly suffer from the following limitations: (1) Many methods achieve multimodal fusion only through simple feature concatenation or addition, lacking effective modeling of the correlation between heterogeneous features, and redundant information interferes with classification decisions; (2) Existing Mamba work mostly focuses on hyperspectral image classification tasks from a single data source, and its application in multimodal fusion classification scenarios is still insufficient. How to overcome the above limitations and achieve efficient fusion of spectral, spatial, and elevation information is the key issue currently facing research on hyperspectral and LiDAR fusion classification. Summary of the Invention

[0004] The technical problem this invention aims to solve is to address the issue that existing hyperspectral and LiDAR image fusion classification methods suffer from insufficient utilization of cross-modal complementary information due to simple feature fusion strategies. This invention proposes a collaborative recognition method for hyperspectral and LiDAR images based on Mamba attention fusion. This method extracts shallow features through bi-branch convolution, performs long-distance modeling with linear complexity using a spatial Mamba module, and guides the fusion module through bi-attention interaction to achieve deep cross-modal interaction. This effectively suppresses redundant information interference and improves the accuracy and robustness of ground feature classification.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A collaborative recognition method for hyperspectral and LiDAR images based on Mamba attention fusion includes: 1) Input hyperspectral image patches and LiDAR image patches are fed into a dual-branch convolutional neural network for shallow feature extraction, outputting bimodal embedding features of uniform dimension. and ; 2) The bimodal embedding features are fed into their respective spatial Mamba modules to perform multi-scale global-local long-distance dependency feature modeling with linear time complexity, generating three scales of deep feature maps for each modality; 3) For bimodal feature maps at various scales, the channel attention module and spatial attention module are executed in parallel to calculate cross-modal attention weights, thereby achieving deep interaction and fusion of multimodal features; 4) After upsampling and aligning the feature maps fused from the three scales, they are stitched together along the channel dimension, flattened, and then output as land cover category prediction results through a fully connected layer. The cross-entropy loss function is used for training supervision.

[0006] Optionally, the detailed steps in step 1) include: the hyperspectral image branch extracts spectral-spatial features through a 3D convolutional layer with a kernel size of 5×1×1, and then performs dimension alignment through a 2D convolutional layer with a kernel size of 1×1, wherein the 3D convolutional layer has 8 filters and the 2D convolutional layer has 64 filters; the LiDAR data branch extracts features through two 2D convolutional layers with a kernel size of 1×1, the first convolutional layer has 32 filters and the second convolutional layer has 64 filters; batch normalization and LeakyReLU activation functions are added after each convolutional layer, and the output is uniformly a bimodal embedding feature with C=64 dimensions. and .

[0007] Optionally, in step 2), the spatial Mamba module consists of a group normalization layer, Mamba blocks, and residual connections, and its forward propagation process is expressed as follows: , ;

[0008] In the above formula, As input features, For group normalization operation, The features are normalized, and Mamba(·) represents the selective state-space model processing procedure. For output features; the Mamba block adopts a selective state-space model (S6), which dynamically adjusts the state transition matrix through a selective scanning mechanism related to the input, and achieves global long-distance dependency modeling in linear time complexity O(N).

[0009] Optionally, in step 2), a multi-scale feature pyramid is constructed by combining three cascaded spatial Mamba modules with two average pooling layers with a stride of 2: the first spatial Mamba module outputs the original resolution P×P feature map; then, after being downsampled by 2 times by an average pooling layer with a pooling window of 2×2 and a stride of 2, the second spatial Mamba module outputs a 1 / 2 resolution P / 2×P / 2 feature map; after being downsampled again by an average pooling layer of the same specification, the third spatial Mamba module outputs a 1 / 4 resolution P / 4×P / 4 feature map, thereby constructing a three-scale feature pyramid.

[0010] Optionally, the detailed steps in step 3) include: 3.1) The channel attention weights are calculated by summing the bimodal feature maps, performing global average pooling, and then passing them through a bottleneck fully connected layer and sigmoid activation to obtain the channel attention weights for each modality. The expression is as follows: , , In the above formula, GAP represents global average pooling. For the bottleneck fully connected layer with a compression ratio r=16, and These are fully connected layers for the hyperspectral and LiDAR branches, respectively, with Sigmoid as the activation function and s as the current scale index; 3.2) Spatial attention maps are obtained by concatenating bimodal feature maps along the channel dimension, performing preliminary fusion via 3×3 convolution, followed by global average pooling and global max pooling, and then passing through convolutional layers and Sigmoid activation to obtain spatial attention weight maps for each modality. The expression is as follows: , , , In the above formula, GMP stands for Global Max Pooling. This is the feature map after the bimodal features are initially fused using 3×3 convolution. and These are 3×3 convolutional layers for hyperspectral and LiDAR branches, respectively, with padding of 1 and stride of 1. 3.3) The fusion feature is obtained by summing the channel attention weights and spatial attention maps applied sequentially to each modality feature. Its expression is: , In the above formula, ⊙ represents element-wise multiplication, and the channel attention weights are broadcast and multiplied along the spatial dimension. is the fusion output feature at the s-th scale.

[0011] Optionally, the detailed steps in step 4) include: 4.1) By using bilinear interpolation upsampling, the fused feature maps of P / 4×P / 4 and P / 2×P / 2 scales are restored to P×P resolution respectively, and then concatenated with the original resolution feature maps in the channel dimension to form multi-scale aggregated features; 4.2) After flattening the multi-scale aggregated features, they are sequentially passed through two fully connected layers for classification mapping, outputting the predicted probability of each category; 4.3) During training, end-to-end supervised training is performed using the cross-entropy loss function. The cross-entropy loss function measures the difference between the probability distribution predicted by the model and the true distribution. The smaller the value, the closer the model's prediction is to the reality. The calculation expression is: , In the above formula, For the true label of class c, this embodiment uses one-hot encoding, that is, if the sample belongs to class c, =1, otherwise =0, Let N be the predicted probability of the model for class c, N be the number of training samples, and C be the total number of land cover categories.

[0012] Due to the real label It is a one-hot vector. In actual calculation, the above formula only calculates the logarithmic loss for the true category and its corresponding terms. Terms that are not true categories are directly ignored because they are multiplied by 0. This mechanism enables the model to accurately calculate the prediction error and apply targeted penalties during backpropagation, thereby efficiently optimizing all network parameters and driving the model to learn a more discriminative multimodal fusion classification representation.

[0013] Compared with existing technologies, this invention has the following advantages: Considering the natural complementarity between hyperspectral images and LiDAR data in terms of spectral features and 3D structural information, and the problems introduced by existing methods such as simple feature stitching leading to redundant interference, this invention constructs a dual-branch convolutional feature extractor to extract shallow spatial-spectral features of the two modalities separately, fully utilizing the feature extraction advantages of 3D convolution in the spectral dimension; it introduces a spatial Mamba module, utilizing its linear time complexity state-space modeling mechanism to perform multi-scale global-local long-distance dependency feature mining on the two modalities at the pixel level, maintaining the ability to perceive the global context; and it designs a dual-attention interactive guided fusion module (DAIGF), which dynamically suppresses redundant information and strengthens the collaborative expression of effective features between modalities through a parallel mechanism of channel attention and spatial attention, achieving deep interaction and fine fusion of cross-modal complementary information. The method of this invention is applicable to various remote sensing application scenarios such as urban planning, disaster monitoring, agricultural surveys, and environmental assessment, and has good classification accuracy and robustness. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below:

[0015] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the overall model architecture of the method in an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the dual-branch convolutional shallow feature extraction module in an embodiment of the present invention.

[0018] Figure 4 This is a schematic diagram of the spatial Mamba deep feature extraction module in an embodiment of the present invention.

[0019] Figure 5 This is a schematic diagram of the dual-attention interactive guidance fusion module in an embodiment of the present invention. Detailed Implementation

[0020] like Figure 1 and Figure 2 As shown, a method for collaborative recognition of hyperspectral and LiDAR images based on Mamba attention fusion includes: 1) Input hyperspectral image patches and LiDAR image patches are fed into a dual-branch convolutional neural network for shallow feature extraction, outputting bimodal embedding features of uniform dimension. and ; 2) The bimodal embedding features are fed into their respective spatial Mamba modules to perform multi-scale global-local long-distance dependency feature modeling with linear time complexity, generating three scales of deep feature maps for each modality; 3) For bimodal feature maps at various scales, the channel attention module and spatial attention module are executed in parallel to calculate cross-modal attention weights, thereby achieving deep interaction and fusion of multimodal features; 4) After upsampling and aligning the feature maps fused from the three scales, they are stitched together along the channel dimension, flattened, and then output as land cover category prediction results through a fully connected layer. The cross-entropy loss function is used for training supervision.

[0021] See Figure 2 It can be seen that step 1) is the dual-modal shallow feature extraction process; step 2) is the spatial Mamba deep feature modeling process; step 3) is the dual attention interactive guided fusion process; and step 4) is the multi-scale feature aggregation and classification output process.

[0022] like Figure 3 As shown, the detailed steps in step 1) include: Before feature extraction, the original hyperspectral images and LiDAR data are preprocessed. Since the physical properties and dimensions of data acquired from different sensors are unrelated, this embodiment performs maximum-minimum normalization on both the hyperspectral images and LiDAR data, linearly mapping the original data to the [0,1] interval to eliminate the influence of dimensional differences, enabling effective comparison and fusion of multimodal features. Subsequently, the normalized data is cropped into image patches of uniform size.

[0023] For hyperspectral image patches First, a 3D convolutional layer with a kernel size of 5×1×1 is used for spectral-spatial joint feature extraction. Because hyperspectral images have a series of continuous bands, they can be considered as a 3D cube (spatial dimensions x and y and spectral dimension λ). The 3D convolution can simultaneously process information from these dimensions, effectively extracting spatial-spectral joint features. This 3D convolutional layer has 8 filters. Subsequently, a 2D convolutional layer with a kernel size of 1×1 is used for dimension alignment, with 64 filters.

[0024] For LiDAR data blocks Since LiDAR data is a single-channel image, two 1×1 two-dimensional convolutional layers are stacked to extract elevation features. The first convolutional layer has 32 filters and the second convolutional layer has 64 filters.

[0025] Each convolutional layer is followed by batch normalization (BN) and the LeakyReLU activation function to accelerate model convergence. After this processing, image patches from both modalities are mapped to a unified C=64-dimensional bimodal embedding feature. and This maintains a consistent number of channels, which facilitates subsequent multimodal feature interaction.

[0026] like Figure 4 As shown, in step 2), the fusion and classification of hyperspectral and LiDAR images is a pixel-level classification task, and the feature representation needs to meet both accuracy and discriminability. This embodiment uses spatial Mamba modules as the basic unit to construct a deep feature extractor, which can achieve powerful long-distance dependency modeling capabilities in linear computational complexity O(N).

[0027] The spatial Mamba module consists of a group normalization layer, Mamba blocks, and residual connections. Its forward propagation process is expressed as follows: , ,

[0028] In the above formula, For input features, GN is the group normalization operation. The features are normalized, and Mamba(·) represents the Mamba block processing procedure. This is the output of the current space Mamba module, with the embedded dimension, i.e. the dimension of the intermediate hidden state, being C=64.

[0029] The Mamba block is a standard Mamba block employing a selective state-space model (S6). It dynamically adjusts the state transition matrix through an input-related selective scanning mechanism, enabling the model to adaptively select relevant information for state updates based on the current input, achieving global long-distance dependency modeling with a linear time complexity of O(N). The convolutional layers within the Mamba block extract local features, while the S6 model processes these features and captures long-term dependencies; the two complement each other. Group normalization and residual connections contribute to the stable learning of the spatial Mamba module.

[0030] Furthermore, the core processing within the Mamba block relies on the Selective State-Space Model (SSM), which maps one-dimensional inputs to hidden states, and then to outputs. Its state equations and output equations can be expressed as: , ,

[0031] In the above formula, For the current time step input, Currently in a hidden state. This is the output. and These are the discretized state transition matrix and the input matrix, respectively. To achieve dynamic selection of input information, Mamba introduces a step size parameter that depends on the input data. Parameters in continuous state and Discretization is performed, and the specific formula is as follows: , ,

[0032] Through the above discretization operation, the step size... The degree to which the model focuses on or ignores the current input can be dynamically controlled: a larger value indicates that the model focuses more on the current information, while a smaller value indicates that it ignores it. This selective mechanism allows the model to filter out irrelevant information (such as redundant spectral noise) and retain key information related to the current ground features over a long period of time.

[0033] This embodiment uses three cascaded spatial Mamba modules combined with two average pooling layers to construct a three-scale feature pyramid: the first spatial Mamba module outputs a feature map of original resolution P×P; then, after being downsampled by a factor of 2 by an average pooling layer with a pooling window of 2×2 and a stride of 2, the second spatial Mamba module outputs a feature map of 1 / 2 resolution P / 2×P / 2; after being downsampled again by an average pooling layer of the same specification, the third spatial Mamba module outputs a feature map of 1 / 4 resolution P / 4×P / 4. This three-scale feature pyramid effectively extracts global and local spatial information of modalities, while utilizing group normalization to batch integrate spatial and spectral information to obtain refined and unique deep features, providing support for subsequent multi-scale fusion.

[0034] like Figure 5 As shown in step 3), inspired by channel attention and spatial attention mechanisms, the Dual Attention Interaction Guided Fusion Module (DAIGF) consists of parallel channel attention and spatial attention branches, which allows redundant information in both spatial and channel dimensions to be suppressed while important information can be enhanced. The deep semantic features of hyperspectral images and LiDAR are simultaneously fed into this module, and attention maps are jointly generated for both modalities, fully utilizing complementary information between modalities rather than generating attention maps for each modality independently.

[0035] Because the average pooling layer following each spatial Mamba module outputs a feature map of one size, feature maps of three scales for each of the two modalities can be obtained after the spatial Mamba module. Taking a hyperspectral image as an example, this yields... (P×P) (P / 2 × P / 2) and (P / 4×P / 4).

[0036] With the highest spatial resolution feature map , For example: 3.1) The channel attention module utilizes the inter-channel relationships between different modalities. Complementary information from the two branches is aggregated through element-wise addition and global average pooling (GAP), compressing the two-dimensional spatial features into a one-dimensional channel representation. To further improve efficiency, channel compression is performed by a fully connected layer (compression ratio r=16), and then the channel dimension is restored by dual fully connected layers corresponding to the two input branches respectively. The channel attention weight map is obtained after Sigmoid activation. The calculation process is as follows: , , In the above formula, To compress the fully connected layer, and The two branches are fully connected for the recovery of hyperspectral images and LiDAR data, respectively. This dual-branch design makes the channel attention weights of the two modalities independent yet mutually aware, implicitly establishing cross-modal channel correlations by sharing a front fully connected layer.

[0037] 3.3) Through the dual-attention interaction-guided fusion module described above, one-dimensional channel attention and two-dimensional spatial attention are finally obtained. The fusion feature map calculation process is as follows: , In the above formula, s represents the scale index of the input feature map. and These represent the input feature maps of hyperspectral images and LiDAR data, respectively. The output fusion feature is represented by ⊙, which indicates element-wise multiplication, where the channel attention feature map is broadcast and multiplied along the spatial dimension.

[0038] Step 4) Perform multi-scale aggregation of the feature maps fused from the three scales for the final land cover classification.

[0039] 4.1) By using bilinear interpolation upsampling, the P / 4×P / 4 and P / 2×P / 2 scale fused feature maps are restored to P×P resolution respectively, and then concatenated with the original resolution P×P feature map in the channel dimension to form a multi-scale aggregated feature. 4.2) After flattening the multi-scale aggregated features, they are sequentially passed through two fully connected layers for classification mapping, outputting the predicted probability of each land cover category; 4.3) During training, this embodiment uses the cross-entropy loss function for end-to-end supervised training of the entire network. The cross-entropy loss function measures the difference between the probability distribution predicted by the model and the true distribution. The smaller the value, the closer the model's prediction is to the reality. The calculation expression is: , In the above formula, This is the true label of class c (one-hot encoding). Let N be the predicted probability of the model for class c, N be the number of training samples, and C be the total number of land cover categories. The backpropagation algorithm is used to jointly optimize all network parameters, driving the model to learn an effective multimodal fusion classification representation. Because It is a one-hot vector. In fact, the formula only calculates the true category and its corresponding terms. Other terms are ignored because they are multiplied by 0.

[0040] To verify the effectiveness of this invention, two publicly available HSI-LiDAR benchmark datasets, the Houston 2013 dataset and the MUUFL dataset, were selected for experiments. The Houston 2013 dataset was collected in June 2012 on the University of Houston campus and surrounding urban areas, provided by the 2013 GRSS Data Fusion Competition. The hyperspectral images were taken by a CASI-1500 sensor, containing 144 spectral bands, with an image size of 349×1905 pixels and a spatial resolution of 2.5 meters. The LiDAR data is single-band, containing 15,029 labeled samples distributed across 15 land cover categories. Twenty labeled pixels from each category were selected as training data, and the remainder were used as test data. The MUUFL dataset was collected on November 8, 2010, at the Gulf Park University campus in South Mississippi, Mississippi, USA. Hyperspectral images were captured by a CASI-1500 sensor, containing 64 spectral bands and a size of 325×220 pixels. LiDAR data was obtained using a Gemini airborne laser topographic mapper, containing 53,687 labeled samples distributed across 11 land cover categories. Twenty samples from each category were selected to construct the training set, with the remaining samples used as the test set. Three quantitative evaluation metrics were used to measure the accuracy of land cover classification results: Overall Accuracy (OA), Average Accuracy (AA), and Kappa coefficient. Higher values ​​for these metrics indicate better model classification performance. All metrics were calculated by averaging the results of ten experiments to ensure fair and reliable comparisons. OA measures the percentage of samples correctly classified by the model out of the total sample size; AA is the average classification accuracy for all individual land cover categories; and the Kappa coefficient comprehensively considers the difference between actual classification results and random classification results, measuring the consistency and reliability of the overall classification results.

[0041] This embodiment combines the proposed method with Fusion_HCT (for details, see Zhao G, Ye Q, Sun L, et al. Jointclassification of hyperspectral and LiDAR data using a hierarchical CNN and transformer[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 61:1-16.), DSHFNet (for details, see Feng Y, Song L, Wang L, et al. DSHFNet: Dynamic scalehierarchical fusion network based on multiattention for hyperspectral image and LiDAR data classification[J]. IEEE Transactions on Geoscience and RemoteSensing, 2023, 61: 1-14.), AMSSE-Net (see Gao H, Feng H, Zhang Y, et al. for details. AMSSE-Net: Adaptive multiscale spatial-spectral enhancement network for classification of hyperspectral and LiDAR data[J]. IEEE Transactions onGeoscience and Remote Sensing, 2023, 61: The method of this invention is compared with six advanced algorithms, including 1-17. Table 1 shows the quantitative comparison results of the method of this invention and the above-mentioned comparison algorithms on the Houston dataset, and Table 2 shows the quantitative comparison results on the MUUFL dataset.

[0042] Table 1 shows the classification performance (%) of different methods on the Houston dataset.

[0043] Table 2 shows the classification performance (%) of different methods on the MUUFL dataset.

[0044] As shown in Table 1, the proposed method in this embodiment outperforms all comparative methods in terms of OA, AA, and Kappa coefficients on the Houston dataset, achieving the best classification performance. Compared with the second-best performing comparative model MICF-Net, the proposed method improves OA / AA / Kappa by 1.00% / 1.65% / 1.08%, respectively. Regarding classification accuracy across categories, the proposed method achieves the highest or second-highest accuracy in eight categories, demonstrating excellent performance in more than half of the 15 categories, with all categories maintaining a stable accuracy above 90%. As shown in Table 2, the proposed method also achieves the best results in OA and Kappa coefficients on the MUUFL dataset, reaching 84.24% and 79.54%, respectively, representing improvements of 0.31% and 0.25% compared to the second-best performing comparative model MICF-Net. Furthermore, the classification map generated by the proposed method is closest to the ground truth, exhibiting clearer edges and fewer noise points, further validating the effectiveness of the proposed method.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hyperspectral and LiDAR image collaborative recognition method based on Mamba attention fusion, characterized in that, include: 1) The input hyperspectral image block and LiDAR image block are respectively sent into a double-branch convolutional neural network for shallow feature extraction, and a unified dimension double-modal embedding feature is output and ; 2) The bimodal embedding features are fed into their respective spatial Mamba modules to perform multi-scale global-local long-distance dependency feature modeling with linear time complexity, generating three scales of deep feature maps for each modality; 3) For bimodal feature maps at various scales, the channel attention module and spatial attention module are executed in parallel to calculate cross-modal attention weights, thereby achieving deep interaction and fusion of multimodal features; 4) After upsampling and aligning the feature maps fused from the three scales, they are stitched together along the channel dimension, flattened, and then output as land cover category prediction results through a fully connected layer. The cross-entropy loss function is used for training supervision.

2. The Mamba attention fusion-based hyperspectral and LiDAR image collaborative recognition method according to claim 1, characterized in that, The detailed steps in step 1) include: 1.1) The hyperspectral image branch extracts spectral-spatial features through a 3D convolutional layer with a kernel size of 5×1×1, and then performs dimension alignment through a 2D convolutional layer with a kernel size of 1×1. The 3D convolutional layer has 8 filters and the 2D convolutional layer has 64 filters. 1.2) The LiDAR data branch is used for feature extraction through two 2D convolutional layers with a kernel size of 1×1. The first convolutional layer has 32 filters and the second convolutional layer has 64 filters. 1.3) Add Batch Normalization and LeakyReLU activation function after each convolutional layer, and output uniform C=64-dimensional dual-modal embedding features and .

3. The Mamba attention fusion based hyperspectral and LiDAR image collaborative recognition method according to claim 1, characterized in that, In step 2), the spatial Mamba module consists of a group normalization layer, Mamba blocks, and residual connections. Its forward propagation process is expressed as follows: ， ， In the above formula, is an input feature, is a group normalization operation, is a normalized feature, is a selective state space model processing procedure, is an output feature; M The amba block employs a selective state-space model, dynamically adjusting the state transition matrix through an input-related selective scanning mechanism to achieve global long-distance dependency modeling in linear time complexity O(N).

4. The Mamba attention fusion-based hyperspectral and LiDAR image collaborative recognition method according to claim 1, characterized in that, In step 2), a multi-scale feature pyramid is constructed by combining three cascaded spatial Mamba modules with two average pooling layers with a stride of 2. Specifically, the first spatial Mamba module outputs the original resolution P×P feature map; then, after being downsampled by 2 times by an average pooling layer with a pooling window of 2×2 and a stride of 2, the second spatial Mamba module outputs a 1 / 2 resolution P / 2×P / 2 feature map; after being downsampled again by an average pooling layer of the same specification, the third spatial Mamba module outputs a 1 / 4 resolution P / 4×P / 4 feature map, thus constructing a three-scale feature pyramid.

5. The Mamba attention fusion based hyperspectral and LiDAR image collaborative recognition method according to claim 1, characterized in that, The detailed steps in step 3) include: 3.1) The channel attention weights are calculated by summing the bimodal feature maps, performing global average pooling, and then passing them through a bottleneck fully connected layer and sigmoid activation to obtain the channel attention weights for each modality. The expression is as follows: ， ， In the above formula, GAP is a global average pooling, is a bottleneck fully connected layer with compression ratio r = 16, and are fully connected layers of hyperspectral and LiDAR branches, respectively, is an activation function, s is the current scale index, and are the calculated channel attention weights of the hyperspectral branch and the LiDAR branch, respectively; 3.2) Spatial attention maps are obtained by concatenating bimodal feature maps along the channel dimension, performing preliminary fusion via 3×3 convolution, followed by global average pooling and global max pooling, and then passing through convolutional layers and Sigmoid activation to obtain spatial attention weight maps for each modality. The expression is as follows: ， ， In the above formula, GMP stands for Global Max Pooling. This is the feature map after the bimodal features are initially fused using 3×3 convolution. and These are 3×3 convolutional layers for the hyperspectral and LiDAR branches, respectively, with padding of 1 and stride of 1 to maintain spatial resolution. and These are the calculated spatial attention weight maps for the hyperspectral and LiDAR branches, respectively. 3.3) The fusion feature is obtained by summing the channel attention weights and spatial attention maps applied sequentially to each modality feature. Its expression is: ， In the above formula, ⊙ represents element-wise multiplication, where the channel attention weights are broadcast and multiplied along the spatial dimension. is the fusion output feature at the s-th scale.

6. The hyperspectral and LiDAR image collaborative recognition method based on Mamba attention fusion according to claim 1, characterized in that, The detailed steps in step 4) include: 4.1) By using bilinear interpolation upsampling, the fused feature maps of P / 4×P / 4 and P / 2×P / 2 scales are restored to P×P resolution respectively, and then concatenated with the original resolution feature maps in the channel dimension to form multi-scale aggregated features; 4.2) After flattening the multi-scale aggregated features, they are sequentially passed through two fully connected layers for classification mapping, outputting the predicted probability of each category; 4.3) During training, end-to-end supervised training is performed using the cross-entropy loss function, the expression of which is: ， In the above formula, For the c-th type of real label, To predict the probability, N is the number of samples and C is the total number of categories.

7. The hyperspectral and LiDAR image collaborative recognition method based on Mamba attention fusion according to claim 1, characterized in that, The method also includes a performance evaluation step, using overall accuracy, average accuracy, and Kappa coefficient as classification performance evaluation metrics, and conducting quantitative comparative experiments on two publicly available hyperspectral-LiDAR benchmark datasets, Houston and MUUFL.

Images