Adaptive fusion of hyperspectral and LiDAR data based on mamba structure for collaborative classification

Abstract

The application provides a hyperspectral and LiDAR data adaptive fusion collaborative classification method based on a Mamba structure. First, the spatial-spectral joint features of HSI data and the elevation semantic information of LiDAR data are extracted by using a double-branch deep feature extraction architecture. Then, feature aggregation and spatial representation optimization are performed by using a spatial context marker. In the feature fusion stage, a double-channel collaborative attention module (DCCAM) based on the Mamba structure is used to capture global dependencies, and parameter sharing is used to ensure the consistency of heterogeneous features. Finally, an adaptive fusion module (AF) is used to effectively integrate multi-source features and enhance the joint representation of information. Compared with existing multi-modal remote sensing image classification algorithms, the method can achieve higher classification accuracy and significantly improve the computational efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal remote sensing image classification, and particularly relates to a hyperspectral and LiDAR data adaptive fusion collaborative classification method based on a Mamba structure. BACKGROUND

[0002] In recent years, with the progress of satellite technology, it has become possible to capture multi-source images from various satellite sensors, and the information fusion application of hyperspectral imaging (HSI) and light detection and ranging (LiDAR) has attracted extensive attention from researchers. HSI obtains the spectral curve of ground objects, which can distinguish different ground objects by spectral differences, and is therefore widely used in environmental monitoring, national defense, precision agriculture, mineral exploration and other fields. However, HSI lacks height information, which makes it impossible to distinguish ground objects with similar spectra but different heights.

[0003] LiDAR can obtain terrain height information and construct digital surface model (DSM) data, providing missing height stereo and geometric information for HSI. As different heterogeneous remote sensing data, LiDAR data and hyperspectral data can effectively make up for the insufficient information of single modal data by making full use of the complementary spectral and spatial details of the two, and help to significantly improve the classification results.

[0004] Although the traditional Transformer model has significant advantages in global dependency modeling and parallel computing, the self-attention mechanism also brings the problem of quadratic computational complexity. Especially when dealing with high-dimensional data such as hyperspectral images, this computational cost increases dramatically, bringing great challenges to the practical application of the model. In recent years, with the rapid development of state space model (SSM) in the field of deep learning, this parallel processing time series modeling method has gradually attracted attention. The computational complexity and memory occupation of SSM are linearly related to the sequence length, which is crucial for efficient training and inference.

[0005] Recently, a new framework based on SSM, Mamba model, has been proposed. Mamba performs well in computational efficiency and feature extraction ability, and outperforms the Transformer model of the same size in multiple audio and text tasks. Its application in the field of remote sensing has also gradually become a research focus. However, the research of Mamba model mainly focuses on single modal remote sensing data processing.

[0006] In summary, in the field of multi-modal remote sensing image classification, existing methods are difficult to balance global dependency modeling and computational efficiency, and the potential of Mamba model in the field of multi-modal classification has not been fully tapped. SUMMARY

[0007] Therefore, the present application aims to provide a hyperspectral and LiDAR data adaptive fusion collaborative classification method based on Mamba structure, which can realize the modeling ability of Transformer while significantly improving the computing efficiency.

[0008] To achieve the above-mentioned purpose, the present application adopts the following technical solution: a hyperspectral and LiDAR data adaptive fusion collaborative classification method based on Mamba structure, comprising the following steps:

[0009] Step S1: data preprocessing stage: data preprocessing is performed on the hyperspectral image HSI data and laser radar LiDAR data, and the original data is converted into an image space block meeting the input requirements of the model;

[0010] Step S2: dual-branch deep feature extraction: a dual-branch deep feature extraction architecture is constructed, which acts on the preprocessed HSI data and LiDAR data respectively; the spatial-spectral joint features of HSI data and the elevation semantic information of LiDAR data are extracted to capture the height difference and terrain undulation features of ground objects in the vertical direction;

[0011] Step S3: spatial context labeling and optimization: through the designed spatial context labeler, the features extracted from the dual-branch are aggregated, and the spatial representation of the features is optimized by comprehensively considering the context information of the features in the spatial neighborhood;

[0012] Step S4: multi-modal Mamba fusion and classification: the feature sequence processed in steps S1-S3 is input into the designed multi-modal Mamba fusion encoder for encoding and fusion, and then the final classification result of the ground object is output through the classification layer.

[0013] Preferably, the data preprocessing process in step S1 is specifically as follows:

[0014] Let HSI data be X H ∈R M×N×B , LiDAR data be Where M and N represent the width and height of the image respectively, B represents the spectral band number of HSI, and C is the channel number of LiDAR data; the real label y of the ground object i ∈{1,2,…,K}, where K is the number of real label categories; a single pixel in the image is represented as x i,j ∈X H ,x i,j =[x i,j,1 ,…,x i,j,B ], i=1,...M, j=1,...,N;

[0015] Using a spatial-spectral joint strategy, with spatial blocks as model input, the HSI data X is first processed in the preprocessing stage. H Perform a min-max normalization operation, and then extract the cube centered at pixel (i, j). Parallel from Li DAR data X L Extract the corresponding spatial image centered at pixel (i, j). The size of the adjacent regions is P×P.

[0016] Preferably, the dual-branch deep feature extraction architecture described in step S2 includes an HSI joint feature extraction module. and LiDAR spatial feature extraction module

[0017] Module It includes a sequentially arranged Conv3D layer, batch normalized BN layer, activation layer, HetConv structure, batch normalized BN layer, activation layer, Conv2D layer, batch normalized BN layer, and activation layer;

[0018] Module It includes Conv2D layers, batch normalized BN layers, activation layers, Conv2D layers, batch normalized BN layers, and activation layers arranged in sequence.

[0019] Preferably, the module In the module, the first two activation layers use the ReLU function, and the last activation layer uses the GELU function; In this design, both activation layers use the GELU function.

[0020] Preferably, the HetConv structure specifically employs two parallel Conv2D layers, one of which performs group convolution and the other performs pointwise convolution, and the outputs of the two convolutional layers are then added element-wise.

[0021] Preferably, the extraction of the spatial-spectral joint features of HSI data and the elevation semantic information of LiDAR data specifically includes:

[0022] An HSI cube of size P×P×B Remodeled into 1×P×P×B, and through modules The kernel size is 3×3×9 and the padding is 1×1×0. The kernel is processed by a Conv3D layer, and then the feature representation X is generated by a batch normalized BN layer and an activation layer. in_1 ;

[0023] Feature X in_1 After being reshaped into P×P×32, the data are passed through two parallel Conv2D layers of the HetConv structure, one of which processes the reshaped feature X.in_1 perform group convolution with kernel size 3x3, group number 8, padding 1x1, another point-wise convolution with kernel size 1x1, group number 1, padding 0x0, and the output of the two convolution layers are added element-wise to obtain the feature representation X in_1 perform group convolution with kernel size 3x3, group number 8, padding 1x1, another point-wise convolution with kernel size 1x1, group number 1, padding 0x0, and the output of the two convolution layers are added element-wise to obtain the feature representation X in_2

[0024] X in_2 be input into a Conv2D layer with kernel size 3x3, and generate a spatial-spectral joint feature sequence X of HSI data through a batch normalization BN layer and an activation layer out_H

[0025] a LiDAR spatial image with size PxPxC Input module perform feature extraction, and the kernel size of the two Conv2D layers is 3x3, and the padding is 1x1x0, and the module generate an elevation feature sequence X of LiDAR data out_L .

[0026] Preferably, the spatial context marker in step S3 specifically performs the following operations: first, flatten the spatial-spectral joint feature sequence X with size PxPx32 into a feature block with size P out_H and the elevation feature sequence X out_L into P 2 x32, then down-sample by average pooling, smooth the local information while reducing noise; then embed the learnable absolute position encoding into the feature block:

[0027]

[0028] wherein is the learnable absolute position encoding, Flatten(·) represents the flattening operation, AP(·) represents the pooling layer, and represent the processed feature sequence, represents element-wise addition.

[0029] Preferably, the multi-modal Mamba fusion encoder in step S4 includes a stackable double-channel collaborative attention module DCCAM based on Mamba structure and an adaptive fusion module AF; wherein the double-channel collaborative attention module DCCAM captures long-distance dependency relationship between the feature sequences and while using parameter sharing to ensure the consistency of information processing of the feature sequences and , and outputs the feature​​ and as the input of the adaptive fusion module AF; the adaptive fusion module AF fuses the two input features by weighting with learnable weights and optimizes the fusion result by layer normalization.

[0030] Preferably, the double-channel collaborative attention module DCCAM specifically performs the following operations:

[0031] The input features are subjected to layer normalization operation, and then processed by a linear layer. After that, the features are split to obtain the backbone features and the auxiliary features for the gating MLP

[0032]

[0033] where i ~ (H, L) represents the modal type, Chunk(·) represents the chunk operation, Linear(·) represents the linear layer, LN(·) represents the layer normalization;

[0034] The backbone features are subjected to I / O relationship modeling by the SSM module and then subjected to layer normalization operation; the auxiliary features are processed by a parameter-shared CNN layer, Share = Conv2D(·), and a SiLU function is selected for activation; the activated features are multiplied element by element with the backbone features subjected to layer normalization operation to obtain weighted backbone features The specific process is represented as follows:

[0035]

[0036] where Share(·) represents the parameter-shared layer, represents element-wise multiplication, represents the backbone features subjected to layer normalization operation, represents the auxiliary features activated by the SiLU function.

[0037] Preferably, the adaptive fusion module AF specifically performs the following operations:

[0038] The adaptive fusion module AF initializes a random vector equal in length to the feature sequence The random vector v is activated by a sigmoid function and weighted with and output by the double-channel collaborative attention module DCCAM. Finally, the weighted features are fused by addition and subjected to layer normalization operation by a softmax function. The specific process is represented as follows:

[0039]

[0040] wherein, softmax(·) represents a softmax function; sigmoid(·) represents a sigmoid function;

[0041] The fused feature X after the normalization operation fused is fed into the classification head to obtain the final classification result.

[0042] Compared with the prior art, the present application has the following beneficial effects:

[0043] In view of the fact that the prior method is difficult to simultaneously consider global dependency modeling and computational efficiency, and the potential of the Mamba model in the field of multi-modal classification has not been fully tapped, the present application constructs a hyperspectral and LiDAR data adaptive fusion collaborative classification network based on the Mamba structure for classification, and compared with the existing multi-modal remote sensing image classification algorithm, the method of the present application can realize higher classification accuracy and significantly improve the computational efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0044] Fig. 1 The network architecture schematic diagram of the preferred embodiment of the present application;

[0045] Fig. 2 The feature extraction process schematic diagram of the preferred embodiment of the present application;

[0046] Fig. 3 The feature fusion process schematic diagram of the preferred embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings.

[0048] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0049] It should be noted that the terms used herein are only for the purpose of describing the specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0050] As Figs. 1-3 shown, the present embodiment provides a hyperspectral and LiDAR data adaptive fusion collaborative classification method based on the Mamba structure, which comprises the following steps:

[0051] Step S1: Data preprocessing stage: Perform data preprocessing on hyperspectral image HSI data and lidar LiDAR data to transform the raw data into image spatial blocks that meet the model input requirements;

[0052] Step S2: Dual-branch depth feature extraction: Construct a dual-branch depth feature extraction architecture, which is applied to the preprocessed HSI data and LiDAR data respectively. For HSI data, the focus is on mining its spatial-spectral joint features; for LiDAR data, the focus is on extracting its elevation semantic information to accurately capture the vertical height differences and topographic relief features of ground features.

[0053] Step S3: Spatial context labeling and optimization: By designing a spatial context labeler, the features extracted from the two branches are aggregated. By comprehensively considering the contextual information of the features in the spatial neighborhood, the spatial representation of the features is further optimized.

[0054] Step S4: Multimodal Mamba fusion and classification: The feature sequences processed in the above steps are input into the designed multimodal Mamba fusion encoder for encoding and fusion, and then the final classification result of the ground objects is output through the classification layer.

[0055] In this embodiment, the data preprocessing process described in step S1 is as follows:

[0056] Let the HSI data be X. H ∈R M×N×B LiDAR data is Where M and N represent the width and height of the image, respectively, B represents the number of spectral bands of the HSI, and C is the number of LiDAR data channels. Ground feature real-world label y i ∈{1,2,…,K}, where K is the number of true label categories. A single pixel in an image can be represented as x. i,j ∈X H ,x i,j =[x i,j,1 ,…,x i,j,B ], B is the number of spectral bands, i = 1,...M, j = 1,...,N.

[0057] The method of this invention uses a spatial-spectral joint strategy, taking spatial blocks as input to the model. In the preprocessing stage, the HSI data X is first processed. H Perform a min-max normalization operation, and then extract the cube centered at pixel (i, j). Parallel from LiDAR data X L Extract the corresponding spatial image centered at pixel (i, j). The size of the adjacent regions is (P×P).

[0058] In this embodiment, the dual-branch deep feature extraction architecture described in step S2 includes an HSI joint feature extraction module. and LiDAR spatial feature extraction module

[0059] Module It includes sequentially arranged Conv3D layers, batch normalized BN layers, ReLU function activation layers, HetConv structures, batch normalized BN layers, ReLU function activation layers, Conv2D layers, batch normalized BN layers, and GELU activation layers;

[0060] The HetConv structure specifically employs two parallel Conv2D layers, one performing group convolution and the other performing pointwise convolution. The outputs of the two convolutional layers are then added element-wise. ).

[0061] Module It includes Conv2D layers, batch normalized BN layers, GELU activation layers, Conv2D layers, batch normalized BN layers, and GELU activation layers arranged in sequence.

[0062] The spatial-spectral joint features of HSI data and the elevation semantic information of LiDAR data are extracted, specifically as follows:

[0063] First, take an HSI cube of size (P×P×B). Remodeled as (1×P×P×B), and through modules The kernel is processed by a Conv3D layer with a kernel size of (3×3×9), and then the feature representation X is generated by a batch normalized BN layer and an activation layer. in_1 To maintain the spatial height and width of the image, the Conv3D layer uses a padding of (1×1×0).

[0064] Next, feature X in_1 After being reshaped into P×P×32, the data are passed through two parallel Conv2D layers of the HetConv structure, one of which processes the reshaped feature X. in_1 Perform a group convolution with a kernel size of 3×3, 8 groups, and 1×1 padding, and another convolution on the reshaped feature X. in_1 Perform pointwise convolutions with a kernel size of 1×1, 1 group, and 0×0 padding, and then add the outputs of the two convolutional layers element-wise. The feature representation X is obtained through a batch normalized BN layer and an activation layer. in_2

[0065] Finally, X in_2The HSI data spatial-spectral joint feature sequence X is input into a Conv2D layer with a kernel size of 3x3, and is generated through a batch normalization BN layer and an activation layer out_H .

[0066] The LiDAR spatial image with a size of PxPx C is flattened into a feature block with a size of (P Input module Feature extraction is performed, and two Conv2D layers with a kernel size of 3x3 and padding of 1x1x0 are used to generate the elevation feature sequence X of the LiDAR data out_L .

[0067] In this embodiment, the spatial context marker in step S3 is processed as follows: first, the spatial-spectral joint feature sequence X with a size of (P out_H x P x 32) and the elevation feature sequence X out_L are flattened into a feature block with a size of (P 2 x 32), and then down-sampling is performed through average pooling to smooth the local information while reducing noise. Then, the learnable absolute position encoding is embedded into the feature block, which enables the model to capture the position information of each element in the input data, thereby better understanding the order or spatial relationship of the data. The above process can be summarized by the following equation:

[0068]

[0069] wherein is the learnable absolute position encoding, Flatten(·) represents the flattening operation, AP(·) represents the pooling layer, and represent the processed feature sequence, denotes element-wise addition.

[0070] In this embodiment, the multi-modal Mamba fusion encoder in step S4 includes a stackable dual-channel collaborative attention module DCCAM (Dual Channel Collaborative Attention Module) based on the Mamba structure and an adaptive fusion module AF (Adaptive Fusion Block). The DCCAM captures long-distance dependencies between HSI and LiDAR information features (feature sequences and ) while ensuring consistency in HSI and LiDAR information processing through parameter sharing, maximizing the learning and utilization of the correlation and complementarity between HSI and LiDAR information, and outputting features and As the input of the adaptive fusion module AF; AF uses learnable weights to weight the fusion of two input features, and optimizes the fusion result through layer normalization.

[0071] In this embodiment, the specific processing process of DCCAM is as follows: first, the input features are subjected to layer normalization operation, and then are processed by using linear layer, and then the features are divided to obtain the backbone features and the auxiliary features for the gated MLP.

[0072]

[0073] Where i ~ (H, L) represents the modal type, Chunk (·) represents the chunk operation, Linear (·) represents the linear layer, and LN (·) represents the layer normalization.

[0074] Then, the SSM module is used to model the I / O (Input and Output) relationship of the backbone features, and then the layer normalization operation is performed, while the auxiliary features are processed by a parameter-shared CNN layer, Share = Conv2D (·). And select SiLU function for activation, multiply the activated features with the backbone features after layer normalization operation element by element to obtain the weighted backbone features The specific process is as follows:

[0075]

[0076] Where Share (·) represents a parameter-shared layer, represents element-by-element multiplication, represents the backbone features after layer normalization operation, and represents the auxiliary features after SiLU function activation.

[0077] In this embodiment, the specific processing process of the AF module is as follows: the AF module initializes a random vector with the same length as the feature sequence The random vector v is activated by the sigmoid function and weighted on the output of the DCCAM module and Finally, the weighted features are fused by addition and subjected to layer normalization operation by the softmax function, which can be represented as:

[0078]

[0079] In the formula, softmax (·) represents the softmax function; sigmoid (·) represents the sigmoid function;

[0080] Finally, the fused feature X after the normalization operation fused is fed into the classification head to obtain the final classification result.

[0081] In summary, in view of the fact that existing methods are difficult to simultaneously consider global dependency modeling and computational efficiency, and the potential of the Mamba model in the field of multi-modal classification has not been fully tapped, the hyperspectral and LiDAR data adaptive fusion collaborative classification network based on the Mamba structure can achieve higher classification accuracy and significantly improve the computational efficiency compared with existing multi-modal remote sensing image classification algorithms.

[0082] The above merely describes preferred embodiments of the present application, and any equivalent changes and modifications made within the scope of the present application should be included in the scope of the present application.

Claims

1. A hyperspectral and LiDAR data adaptive fusion collaborative classification method based on Mamba structure, characterized in that: The method comprises the following steps: Step S1: data preprocessing stage: data preprocessing is performed on hyperspectral image HSI data and laser radar LiDAR data, and the original data is converted into an image space block meeting the input requirements of the model; Step S2: dual-branch deep feature extraction: a dual-branch deep feature extraction architecture is constructed, which acts on the preprocessed HSI data and LiDAR data respectively; spatial-spectral joint features of the HSI data and elevation semantic information of the LiDAR data are extracted to capture the height difference and terrain undulation features of the ground objects in the vertical direction; Step S3: spatial context labeling and optimization: through a designed spatial context labeler, the features extracted from the dual-branch are subjected to an aggregation operation, and the spatial representation of the features is optimized by comprehensively considering the context information of the features in the spatial neighborhood; Step S4: multi-modal Mamba fusion and classification: the feature sequence processed in steps S1-S3 is input into a designed multi-modal Mamba fusion encoder for encoding and fusion, and then the final classification result of the ground object is output through a classification layer.

2. The Mamba structure based hyperspectral and LiDAR data adaptive fusion collaborative classification method according to claim 1, characterized in that, The data preprocessing process in step S1 is specifically as follows: Let HSI data be X H ∈R M×N×B , LiDAR data be where M and N represent the width and height of the image respectively, B represents the spectral band number of HSI, C is the channel number of LiDAR data; the real label y of the ground object i ∈{1, 2,..., K}, where K is the number of real label categories; a single pixel in the image is represented as x i，j ∈X H , x i，j =[x i,j,1 ,..., x i，j，B ], i = 1,..., M, j = 1,..., N; The strategy of spectral-spatial joint is used, and a spatial block is taken as an input of the model. In the preprocessing stage, maximum-minimum normalization operation is performed on the HSI data X H , and then a cube centered at pixel (i, j) is extracted from X L The corresponding spatial image centered at pixel (i, j) is extracted from the LiDAR data X , where the size of the adjacent region is P x P.​ 3. The Mamba structure based hyperspectral and LiDAR data adaptive fusion collaborative classification method according to claim 2, characterized in that, The double-branch depth property extraction architecture described in step S2 includes an HSI joint feature extraction module and a LiDAR spatial feature extraction module Module includes sequentially arranged Conv3D layer, batch normalization BN layer, activation layer, HetConv structure, batch normalization BN layer, activation layer, Conv2D layer, batch normalization BN layer and activation layer; Module includes sequentially arranged a Conv2D layer, a batch normalization BN layer, an activation layer, a Conv2D layer, a batch normalization BN layer, and an activation layer.

4. The Mamba structure based hyperspectral and LiDAR data adaptive fusion collaborative classification method according to claim 3, characterized in that, The module In the module, the first two activation layers adopt ReLU functions, and the last activation layer adopts a GELU function. In the module, the two activation layers adopt GELU functions.

5. The Mamba structure based hyperspectral and LiDAR data adaptive fusion collaborative classification method according to claim 4, characterized in that, The HetConv structure is specifically: two parallel Conv2D layers are adopted, one of which performs group convolution, and the other performs point-by-point convolution, and the outputs of the two convolution layers are subjected to an element-by-element addition operation.

6. The Mamba structure-based hyperspectral and LiDAR data adaptive fusion collaborative classification method according to claim 5, characterized in that, The spatial-spectral joint features of the HSI data and the elevation semantic information of the LiDAR data are extracted, specifically as follows: An HSI cube of size P×P×B Remodeled into 1×P×P×B, and through modules The kernel size is 3×3×9 and the padding is 1×1×0. The kernel is processed by a Conv3D layer, and then the feature representation X is generated by a batch normalized BN layer and an activation layer. in_1 ; Reshape the feature X in_1 Reshape the feature X in_1 Perform group convolution with kernel size 3x3, group number 8, and padding 1x1 in_1 Perform point-wise convolution with kernel size 1x1, group number 1, and padding 0x0 in_2 ; X in_2 The HSI data is input into a Conv2D layer with a kernel size of 3x3, and a spatial-spectral joint feature sequence X of the HSI data is generated through a batch normalization BN layer and an activation layer out_H ; a LiDAR spatial image of size P x P x C input module Feature extraction is performed, the kernel size of two Conv2D layers is 3 x 3, and the padding is 1 x 1 x 0, the module generate the elevation feature sequence X of the LiDAR data out_L .

7. The Mamba structure based hyperspectral and LiDAR data adaptive fusion collaborative classification method according to claim 1, characterized in that, The spatial context marker described in step S3 specifically performs the following operations: First, it generates a spatial-spectral joint feature sequence X of size P×P×32. out_H and elevation feature sequence X out_L The cube is flattened into P 2 The feature blocks are divided into 32×32 blocks, and then downsampled using average pooling to smooth local information while reducing noise. The learnable absolute position encoding is then embedded into the feature blocks. wherein is a learnable absolute position encoding, Flatten(·) represents a flattening operation, AP(·) represents a pooling layer, and represents a processed feature sequence, denotes element-wise addition.

8. The Mamba structure based hyperspectral and LiDAR data adaptive fusion collaborative classification method according to claim 1, characterized in that, The multi-modal Mamba fusion encoder in step S4 comprises a stackable dual-channel collaborative attention module DCCAM and an adaptive fusion module AF based on the Mamba structure; wherein the dual-channel cooperative attention module DCCAM performs long-distance dependency capturing on the feature sequence and while using parameter sharing to ensure consistency of information processing of the feature sequence and and outputs the feature and as an input of the adaptive fusion module AF; The adaptive fusion module AF utilizes learnable weights to perform weighted fusion on the two input features, and optimizes the fusion result through layer normalization.

9. The Mamba structure based hyperspectral and LiDAR data adaptive fusion collaborative classification method according to claim 8, characterized in that, The dual-channel collaborative attention module DCCAM specifically performs the following operations: performing a layer normalization operation on the input features and processing the features using a linear layer, and then splitting the features to obtain backbone features and auxiliary features for the gated MLP Where i~(H, L) represents the modal type, Chunk(·) represents the block operation, Linear(·) represents the linear layer, and LN(·) represents the layer normalization; After the I / O relationship modeling of the backbone features by the SSM module, layer normalization operation is performed; while the auxiliary features are processed by a parameter-shared CNN layer, Share = Conv2D(·), and the SiLU function is selected for activation; the activated features are multiplied element by element with the backbone features after the layer normalization operation to obtain the weighted backbone features The specific process is represented as follows: where Share(·) represents a parameter sharing layer, denotes element-wise multiplication, denotes the backbone feature after the layer normalization operation, denotes the auxiliary feature after the SiLU function activation.

10. The Mamba structure based hyperspectral and LiDAR data adaptive fusion collaborative classification method according to claim 9, characterized in that, The adaptive fusion module AF specifically performs the following operations: The adaptive fusion module AF initializes a random vector equal to the length of the feature sequence The random vector v is activated by a sigmoid function and weighted by the output of the double-channel collaborative attention module DCCAM and The last weighted features are fused by addition and normalized by a softmax function, and the specific process is represented as: In the formula, softmax(·) represents the softmax function; sigmoid(·) represents the sigmoid function; The fused feature X after normalization operation fused is fed into the classification head to obtain the final classification result.

Images