Multi-modal hyperspectral image collaborative classification method and system for HSI and LiDAR data

Abstract

The invention belongs to the technical field of remote sensing image processing, and particularly relates to a multi-modal hyperspectral image collaborative classification method and system oriented to HSI and LiDAR data, by providing a multi-modal Mamba spectral space fusion network model (MMSSF-Net), fusion of multi-scale local space neighborhood features and global spectral sequence features is achieved, and the multi-modal hyperspectral image collaborative classification method and system oriented to HSI and LiDAR data are obtained. And meanwhile, the performance of the classifier is improved by utilizing global-local multi-modal complementary information. The MMSSF-Net model mainly comprises three core components: a multi-modal feature learning module (MMFL), a multi-scale aggregation CNN structure is adopted, multi-modal and multi-scale features are fused, and the learning ability of local spatial features is enhanced; a multi-modal feature fusion module (MMFF) which performs forward and backward selective state space scanning by means of bidirectional Mama, so that the feature of each position can simultaneously fuse global information in front of and behind the context of the feature, thereby further enhancing the fusion effect of the multi-modal complementary information and improving the classification performance; and a decision classification module.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, specifically to a multimodal hyperspectral image collaborative classification method and system for HSI and LiDAR data. Background Technology

[0002] Hyperspectral images (HSI) are high-dimensional images containing a large number of spectral bands. Their core characteristic is their ability to simultaneously reflect both spatial and spectral information of ground features, making them widely used in geological exploration, urban development planning, forest management, and precision agriculture. In contrast, LiDAR data possesses two key advantages: firstly, it can accurately record the elevation information of ground features; and secondly, it has strong resistance to environmental interference. In recent years, how to efficiently construct feature extraction frameworks and integrate HSI and LiDAR data to improve the accuracy of land cover classification has become a research hotspot in the field of remote sensing imagery.

[0003] Traditional multi-source data collaborative classification methods typically employ manually designed feature fusion strategies, such as fusing morphological contour features with raw spectral data or combining feature-level and decision-level approaches. While these methods improve classification accuracy to some extent, their feature extraction process relies excessively on prior knowledge, making it difficult to adaptively capture the essential relationships between multimodal data.

[0004] With the development of deep learning technology, multimodal fusion methods based on convolutional neural networks (CNNs) have gradually become mainstream. These methods automatically extract deep features from HSI and LiDAR data using CNNs and fuse these features through concatenation or weighting, significantly improving classification performance. However, due to the significant differences in spatial distribution and semantic features between HSI and LiDAR data, simple feature concatenation or fusion can easily lead to information redundancy or loss of key features, and it is difficult to effectively capture the deep relationship between the two types of data.

[0005] To more effectively achieve multimodal feature fusion, researchers have proposed fusion methods based on attention mechanisms. For example, the existing technique CN116486251A extracts spatial features from HSI and LiDAR through multi-scale convolution, and uses an attention mechanism to weightedly fuse these multi-scale features. It then introduces spatial topological information from the LiDAR to constrain the fused features, thereby improving intra-class consistency and inter-class separability. Furthermore, CN120510414A proposes a bidirectional masking method based on a mutually guided attention mechanism. This method guides hyperspectral feature learning using LiDAR elevation information, while simultaneously using hyperspectral spectral information to guide LiDAR feature learning. Deep fusion is then achieved through cross-modal dual interactive attention, significantly improving classification accuracy. Although the Transformer can effectively model global dependencies, its secondary computational complexity leads to high training and inference overhead, making it unsuitable for large-scale remote sensing data processing. However, this method still has the following shortcomings: (1) Existing methods focus on spectral, elevation and texture features, ignoring the edge contour information contained in LiDAR data, resulting in limited classification accuracy in detail categories such as road and building boundaries; (2) Although Mamba is more computationally efficient than Transformer, Mamba still faces two major challenges: unidirectional modeling and lack of location information awareness; (3) Existing methods mostly use fixed weighting or simple splicing, which fail to fully explore the hierarchical complementary information of multimodal data. Existing CNN-based local feature extraction modules still have room for improvement in multi-scale feature fusion and discriminative feature enhancement.

[0006] Therefore, there is an urgent need in this field for a multimodal hyperspectral image collaborative classification method that can effectively utilize edge contour information, achieve long-range dependency modeling while maintaining efficient computation, and improve classification accuracy through multi-level fusion strategies, so as to systematically solve the above problems. Summary of the Invention

[0007] In view of this, the present invention aims to overcome the shortcomings of existing collaborative classification techniques for hyperspectral images (HSI) and lidar (LiDAR) data, particularly the insufficient utilization of edge information, the lack of Mamba unidirectional modeling and location information perception, the limited ability to extract local features, and the single fusion strategy, and provides a multimodal hyperspectral image collaborative classification method for HSI and LiDAR data.

[0008] To achieve the above objectives, the present invention provides the following technical solution: Firstly, a multimodal hyperspectral image collaborative classification method for HSI and LiDAR data is provided, specifically including the following steps: S1: Acquire hyperspectral image data and lidar elevation data; apply a gradient joint algorithm to the lidar elevation data to calculate the gradient magnitude of each pixel. As an edge contour intensity map, it thus constitutes a structure containing spectral information. Elevation information and edge contour information The three-modal input dataset.

[0009] Furthermore, the specific calculation process of the gradient joint algorithm is as follows: For a two-dimensional lidar elevation image, the gradient vector of the internal pixels in the row n and column m directions is calculated using the two-point difference method, and the gradient vector of the boundary points is calculated using the forward or backward difference method; the gradient magnitude of each pixel in the edge contour intensity map... It is calculated by the following formula: ; in, Let x be the partial derivative of pixel x with respect to the scalar function f, and i and j represent the row and column, respectively.

[0010] S2: Perform spatial multi-scale patching on the trimodal input data to obtain multi-scale input data. , and Multi-scale features of each input are extracted using a multi-scale aggregation convolutional encoder (MACE), and then fused using an adaptive linear weighting method to obtain multi-scale fused features. , and The multi-scale fused features are input into the spatial attention-based grouping dimensionality reduction module SA-GDR. After grouping, attention enhancement, and dimensionality reduction, low-dimensional local spatial features are output. .

[0011] Furthermore, the processing procedure of the multi-scale aggregation convolutional encoder (MACE) is as follows: (1) The multi-scale input data are respectively processed by convolution functions to obtain encoded features; (2) Adaptive linear weighted fusion is performed on the encoded features to obtain fused features. The calculation formula involved is as follows: ; Where s=1, 2, 3 represent scale indices, corresponding to three spatial scales: λ×λ, 2λ×2λ, and 3λ×3λ, respectively; t=1, 2, 3 represent the data source indexes, corresponding to HSI (t=1), LiDAR (t=2), and edge data (t=3) respectively. Let be the learnable fusion weights, representing the importance of the t-th data source at the s-th scale, and satisfy ... ; The feature transformation function is scale-dependent and data source-dependent; This represents the feature block of the t-th data source at the s-th scale; The output corresponds to scale s for the fused multimodal features.

[0012] S3: Integrate the multi-scale features , and Perform convolution to obtain the spectral space. Angle of elevation and edge contour The high-level semantic features from different modalities are fused and then input into the Mamba module for long-range dependency modeling, ultimately resulting in global sequence features. : ; in, and These are learnable parameters.

[0013] Furthermore, the bidirectional Mamba module is based on a selective state-space model, and the module takes a sequence as input. and output sequence The relationship between them is: ; in It is a Mamba function, specifically: ; in, Indicates the SiLU activation function; , and Represents linear, normalized, and convolutional functions; This represents element-wise multiplication; This indicates a residual connection.

[0014] S4: Apply the low-dimensional local spatial features obtained in step S2 and the global sequence features obtained in step S3 The local feature prediction probability distribution and the global feature prediction probability distribution are obtained by inputting the data into independent classifiers. The two probability distributions are then weighted and averaged to obtain the final fused prediction probability distribution. Based on this distribution, the land cover category label for each pixel is determined, thus completing the classification.

[0015] Furthermore, the specific operation of the weighted average is as follows: ; in, and Predict probability distributions for global features and local features. This is an adjustable parameter.

[0016] Secondly, a multimodal hyperspectral image collaborative classification system for HSI and LiDAR data is provided, including: The data acquisition and preprocessing module is used to acquire hyperspectral images and lidar data, and apply the gradient joint algorithm GJA to extract image edge contours.

[0017] The Multimodal Local Feature Learning Module (MMFL) is configured with a Multi-Scale Aggregated Convolutional Encoder (MACE) to extract and fuse multimodal multi-scale local spatial features. Specifically, the MACE is responsible for extracting multi-scale local spatial features and performing adaptive linear weighted fusion on the multimodal features. The Multimodal Feature Fusion Module (MMFF) is configured with a bidirectional Mamba module for long-range dependency modeling and deep fusion of high-level semantic features from different modalities.

[0018] The decision classification module is used to perform decision-level fusion of local and global features and output the final classification result.

[0019] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of the present invention.

[0020] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described in this invention.

[0021] The beneficial effects of this invention are as follows: (1) This invention relies on the powerful local feature extraction capability of convolutional neural networks (CNN) and the long-distance dependency modeling advantage of the Mamba model. This study organically combines the two and applies them to the joint classification task of HSI and LiDAR. The study proposes the MMSSF-Net network model, which can realize the fusion of multi-scale local spatial neighborhood features and global spectral sequence features, and at the same time improve the classifier performance by utilizing global-local multimodal complementary information.

[0022] (1) This invention extracts edge contour information from LiDAR data as the third modality input through the gradient joint algorithm GJA, which effectively makes up for the lack of information in the classification of detailed features such as roads and buildings in existing methods, and significantly improves the model's ability to perceive and distinguish geometric structures. (2) This invention designs a feature learning framework that combines a multi-scale aggregated convolutional encoder (MACE) with a spatial attention-based grouping dimensionality reduction module (SA-GDR). MACE achieves effective fusion of multi-modal and multi-scale local features through adaptive linear weighting, thereby improving the performance of local feature extraction.

[0023] (3) This invention introduces the long-distance dependency modeling advantage of the bidirectional Mamba model. Based on the selective state-space model, this model performs forward and backward selective scanning of the sequence, enabling each feature position to simultaneously fuse global context information from both its preceding and following sides. While maintaining the same global modeling capability as the Transformer, its computational complexity is reduced from quadratic to linear, significantly improving the efficiency and modeling performance of the model when processing high-dimensional remote sensing data.

[0024] (4) This invention constructs an adaptive fusion hierarchical framework of “local feature learning MMFL → global sequence fusion MMFF → decision fusion”. Learnable adaptive weighting is introduced in feature fusion, and adjustable parameters are used in decision fusion, so that the model can fully explore information at different scales and different modalities, thereby achieving classification results that are superior to existing mainstream methods on multiple public datasets.

[0025] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a technical approach for multimodal hyperspectral image collaborative classification of HSI and LiDAR data; Figure 2 This is a structural diagram of a multimodal hyperspectral image collaborative classification method for HSI and LiDAR data; Figure 3 This is the classification graph result of this invention on the Huston dataset. Detailed Implementation

[0027] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0028] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0029] Example 1: A multimodal hyperspectral image collaborative classification method for HSI and LiDAR data (Multimodal Mamba Spectra-Spatial Fusion Network, MMSSF-Net), mainly consisting of three core components: The system comprises a Multi-modal Feature Learning (MMFL) module, a Multi-modal Feature Fusion (MMFF) module, and a decision classification module. The MMFL module employs a multi-scale aggregation CNN structure, designed to enhance the learning ability of local spatial features through multi-modal, multi-scale feature fusion. The MMFF module leverages Mamba's ability to capture long-range dependencies of multi-modal features, combining local spatial convolutional encoded features with global spectral sequence encoded features to further strengthen the fusion effect of complementary multi-modal information, thereby improving classification performance.

[0030] 1.1 Gradient Joint Algorithm (GJA) for Edge Contour Extraction Hyperspectral images contain rich spectral-spatial information, while lidar data provides elevation information and potential edge contour information of ground features. The complementary features of these two modalities help improve classification performance, especially for ground feature categories with similar spectral responses but different elevations. However, existing methods often neglect edge contour information in lidar data, which is crucial for distinguishing categories with obvious geometric structures such as roads and buildings. To enhance the diversity of input features, this invention first employs a Gradient Joint Algorithm (GJA) to extract edge contour information from lidar elevation data, serving as the input for the third modality.

[0031] Gradients can characterize the intensity changes of each pixel in an image. By combining the elevation information of LiDAR data and using the first-order gradient change to extract the edge contour information of the ground feature distribution, it will help to further improve the ability to distinguish ground features, especially the sample differentiation effect between the shadow area and the non-shadow area of ​​buildings.

[0032] For a two-dimensional discrete image, its gradient is a vector, whose components are the partial derivatives of the function in each direction. For scalar functions... Its gradient can be expressed as follows: ; in, Representation function The partial derivative in the k-th direction. In gradient calculation of a two-dimensional image, both interior and boundary points must be considered. This embodiment uses the central difference method to calculate the gradient of interior points, and its expression is: ; Where h represents the distance between sampling points. Since the sampling points are uniformly distributed, equation (2) simplifies to: ; The row and column directions are denoted by n and m, respectively. The gradients along the n and m directions are calculated using the two-point difference method, and their expressions are as follows: ; For image boundary points, since there may be a lack of sufficient neighboring points, the gradient is calculated using the forward or backward difference method, and its expression is as follows: ; in, For the forward difference result, or This is the result of backward difference.

[0033] To comprehensively reflect the changes in terrain along the row and column directions, the gradient magnitude of each pixel is calculated. The gradient magnitude effectively characterizes the combined gradient changes in the n and m directions. Therefore, the formula for calculating GJA is expressed as: ; in, Let x be the partial derivative of pixel x with respect to the scalar function f, and i and j represent the row and column, respectively.

[0034] 1.2 Multimodal Feature Learning Module (MMFL) The MMFL module designs a multi-scale aggregation CNN for extracting and fusing multimodal, multi-scale local features from HSI and LiDAR data. This model consists of a Multi-Scale Aggregation CNN encoder (MACE) and a Spatial Attention-based Grouped Dimensionality Reduction (SA-GDR) module. MACE is responsible for extracting multi-scale local spatial features and adaptively linearly weighting and fusing the multimodal features. This design fully utilizes multimodal and multi-scale local spatial features while reducing the dimensionality of the fused features, thereby improving computational efficiency and facilitating subsequent global spectral feature extraction. Details are as follows: 1.2.1 Multi-scale Aggregated Convolutional Encoder (MACE) Considering that convolutional kernels of different sizes have different receptive fields and can capture feature information of different granularities, MACE adopts a multi-scale input and adaptive weighted fusion strategy, including a convolutional encoding part and an adaptive linear weighting operation, such as... Figure 1 As shown.

[0035] First, set the basic size parameter λ of the input data. The input to MACE consists of cubes surrounding the HSI target pixels, LiDAR target pixels, and contour image target pixels, with spatial sizes of λ×λ, 2λ×2λ, and 3λ×3λ, respectively.

[0036] Secondly, the encoding part is divided into three blocks, the first two blocks containing kernel sizes of... The second block consists of two-dimensional convolution, batch normalization (BN), and Rectified Linear Unit (ReLU) activation operations. The third block comprises a convolutional kernel layer and a ReLU activation function layer. The number of convolutional kernels is set to 32 and 64.

[0037] To further enhance the CNN's ability to capture detailed information, the input data is processed before multi-scale feature extraction. Perform spatial multi-scale patch segmentation to obtain and ,in, Indicates the scaling factor. Indicates the first The spatial size of each scaling factor. Then, the obtained data is sent to MACE to obtain... , and .

[0038] The MACE process is represented as: ; Where s=1, 2, 3 represent scale indices, corresponding to three spatial scales: λ×λ, 2λ×2λ, and 3λ×3λ, respectively; t=1, 2, 3 represent the data source indexes, corresponding to HSI (t=1), LiDAR (t=2), and edge data (t=3) respectively. Let be the learnable fusion weights, representing the importance of the t-th data source at the s-th scale, and satisfy ... ; The feature transformation function is scale-dependent and data source-dependent; This represents the feature block of the t-th data source at the s-th scale; The output corresponds to scale s for the fused multimodal features.

[0039] 1.3 Multimodal Feature Fusion Module (MMFF) MMFF aims to leverage Mamba's powerful long-range dependency modeling capabilities and linear computational complexity to deeply fuse high-level semantic features from different modalities.

[0040] The Mamba framework is an improved version of the State Space Model (SSMs), and its construction is as follows: Figure 1 As shown. The concept of SSM originated from the continuous linear time-invariant system (LTI). Although it has linear time complexity, it is difficult to capture long-range dependencies.

[0041] Therefore, Mamba designed a selective SSM (S6) to overcome this limitation. Notably, the mapping matrix in S6 depends on the input sequence, allowing selective attention to each input unit. Specifically, the mapping matrix is ​​determined by the input sequence... It can be dynamically calculated , and Input sequence and output sequence The relationship between them can be represented as follows: ; in It is a Mamba function, specifically: ; in, Indicates the SiLU activation function; , and Represents linear, normalized, and convolutional functions, with a kernel size of . ; This represents element-wise multiplication; This indicates a residual connection.

[0042] The structure of MMFF is as follows Figure 1 As shown. First, for the three input data... , and Perform convolution to obtain the spectral space. Angle of elevation and edge contour Advanced features. Next, Blend into features and In the first step, features from different modalities are combined. Then, these three results are fed into Mamba to fully explore the feature representation capabilities within and between modalities. Furthermore, the elevation and edge contour branch data are weighted using two learnable parameters, respectively. Finally, three high-level semantic features are further fused. The MMFM process is represented as follows: ; ; in, This represents the output of the fused MMFM model; and These are learnable parameters.

[0043] Example 2: MMSSF-Net Model Training and Experiment Setup 2.1 Configure the experimental environment The server hardware and software configuration for model training in this embodiment is as follows: graphics card model is GeForce RTX 4090 GPU, processor is Xeon (R) Gold 6430, video memory is 24G, operating system is Ubuntu 20.04, programming language is Python 3.8.1, deep learning framework is PyTorch 1.8.0, and CUDA version is 12.1. In practice, the Adam optimizer is used for network training, with a decay factor of 0.9. The batch size, initial learning rate, and number of training sessions are set to 64, 0.0005, and 500, respectively. The batch size is set to 64, and the training cycle is set to 500.

[0044] To objectively reflect classification performance, different evaluation metrics are used, including accuracy for each category, overall accuracy (OA), average accuracy (AA), and the Kappa coefficient.

[0045] 2.2 Introduction to the Dataset The Houston dataset comprises scenes collected by a Compact Airborne Spectroscopic Imager (CASI) over the University of Houston campus and surrounding urban areas, including digital surface model (DSM) data based on HSI and LiDAR. It has a spatial size of 345 × 1905 pixels and a spatial resolution of 2.5 meters. The HSI data contains 144 spectral bands covering a wavelength range of 0.38–1.35 µm. Furthermore, the Houston dataset includes 15,029 labeled samples representing 15 different land cover categories.

[0046] 2.3 Ablation Experiment To verify the effectiveness of Mamba fusion and edge data in the method, an ablation experiment was conducted in this embodiment. Baseline1 represents the input of HSI and LiDAR data to the MMFL, and Baseline2 represents the input of HSI, LiDAR, and Edge data to the MMFL.

[0047] The experimental results are shown in Table 1. On the Huston dataset, the OA of Baseline1+Mamba is 7.02% higher than that of Baseline1. Figure 3 The classification results and true labels on the Huston dataset are shown.

[0048] Table 1. Contributions of MMSSF-Net Components

[0049] In summary, this invention employs the Gradient Joint Algorithm (GJA) to acquire edge contour data from LiDAR as one of the inputs to a multimodal feature extractor, overcoming the shortcomings of previous studies that suffered from low classification accuracy for roads and buildings due to neglecting edge information. By designing multi-scale feature extraction and multimodal linear weighted fusion, the local feature learning capability of CNNs is improved, replacing the existing CNN-based multimodal multi-scale local spatial feature learning module. Finally, a bidirectional Mamba fusion module is designed to obtain long-distance dependencies within and between modalities, and through decision fusion, global-local prediction information is integrated to achieve more accurate classification.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal hyperspectral image collaborative classification method for HSI and LiDAR data, characterized in that, Includes the following steps: S1: Acquire hyperspectral images and lidar elevation data; apply a gradient joint algorithm to the lidar elevation data to calculate the gradient magnitude of each pixel. As an edge contour intensity map, it thus constitutes a structure containing spectral information. Elevation information and edge contour information A trimodal input dataset; S2: Perform spatial multi-scale patching on the trimodal input data to obtain multi-scale input data. , and ; Multi-scale features of each input are extracted using a multi-scale aggregation convolutional encoder (MACE), and then fused using an adaptive linear weighting method to obtain multi-scale fused features. , and ,Will As a local spatial feature ; S3: Integrate the multi-scale features , and Perform convolution to obtain the spectral space. Angle of elevation and edge contour Advanced semantic features; After fusing high-level semantic features from different modalities, the data is input into a bidirectional Mamba module for long-range dependency modeling, and finally fused to obtain global sequence features. : ； in, and These are learnable parameters; S4: Apply the low-dimensional local spatial features obtained in step S2 and the global sequence features obtained in step S3 The local feature prediction probability distribution and the global feature prediction probability distribution are obtained by inputting the data into independent classifiers. The two probability distributions are then weighted and averaged to obtain the final fused prediction probability distribution. Based on this distribution, the land cover category label for each pixel is determined, thus completing the classification.

2. The method according to claim 1, characterized in that, The specific calculation process of the gradient joint algorithm in step S1 is as follows: For a two-dimensional lidar elevation image, the gradient vector of the internal pixel points in the row n and column m directions is calculated using the two-point difference method, and the gradient vector of the boundary points is calculated using the forward or backward difference method; the gradient magnitude of each pixel point in the edge contour intensity map... It is calculated by the following formula: ； in, Let x be the partial derivative of pixel x with respect to the scalar function f, and i and j represent the row and column, respectively.

3. The method according to claim 1, characterized in that, In step S2, the processing procedure of the multi-scale aggregated convolutional encoder (MACE) is as follows: First, the multi-scale input data are processed by convolution functions to obtain encoded features; then, the encoded features are adaptively linearly weighted and fused to obtain fused features, and the calculation formula involved is as follows: ； Where s=1, 2, 3 represent scale indices, corresponding to three spatial scales: λ×λ, 2λ×2λ, and 3λ×3λ, respectively; t=1, 2, 3 represent the data source indexes, corresponding to HSI (t=1), LiDAR (t=2), and edge data (t=3) respectively. Let be the learnable fusion weights, representing the importance of the t-th data source at the s-th scale, and satisfy ... ; The feature transformation function is scale-dependent and data source-dependent; This represents the feature block of the t-th data source at the s-th scale; The output corresponds to scale s for the fused multimodal features.

4. The method according to claim 1, characterized in that, In step S3, the bidirectional Mamba module is based on a selective state-space model, and the module's input sequence is... and output sequence The relationship between them is: ； in It is a Mamba function, specifically: ； in, Indicates the SiLU activation function; , and Represents linear, normalized, and convolutional functions, with a kernel size of . ; This represents element-wise multiplication; This indicates a residual connection.

5. The method according to claim 1, characterized in that, In step S4, the specific operation of the weighted average is as follows: ； in, and Predict probability distributions for global features and local features. This is an adjustable parameter.

6. A multimodal hyperspectral image collaborative classification system for HSI and LiDAR data that implements the method of any one of claims 1-5, characterized in that, include: The data acquisition and preprocessing module is used to acquire hyperspectral images and lidar data, and apply the gradient joint algorithm GJA to extract image edge contours; The Multimodal Local Feature Learning Module (MMFL) is used to extract and fuse multimodal, multi-scale local spatial features. The Multimodal Feature Fusion Module (MMFF) is configured with a bidirectional Mamba module for long-range dependency modeling and deep fusion of high-level semantic features from different modalities. The decision classification module is used to perform decision-level fusion of local and global features and output the final classification result.

7. A multimodal hyperspectral image collaborative classification system for HSI and LiDAR data according to claim 6, characterized in that, The MACE is responsible for extracting multi-scale local spatial features and performing adaptive linear weighted fusion of multi-modal features.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images