A multi-modal remote sensing classification method fusing cross-scale feature refinement and hybrid state space model

Abstract

The application discloses a kind of multi-modal remote sensing classification methods of fusion cross-scale feature refinement and mixed state space model, to solve the problem of insufficient multi-scale spatial information capture, difficult alignment of heterogeneous features and low classification decision reliability of hyperspectral and laser radar data.The method constructs a double-branch multi-scale adaptive fusion framework, including three core modules of cross-modal global spatial attention, cross-scale feature interaction and state space model sequence fusion.Through global spatial correlation modeling and triangular circulation interaction mechanism, the spectral-spatial-elevation features of multi-modal and multi-scale are fully extracted and aligned;Using the selective scanning mechanism of state space model, efficient long program sequence dependence modeling is realized;And combining contrast learning and Bayesian inference, the feature distribution and decision weight are optimized.Experimental results show that the method has high classification accuracy, generalization ability and robustness in complex ground object classification tasks.

Description

Technical Field

[0001] This invention belongs to the field of multimodal remote sensing image processing and computer vision technology, specifically involving a multimodal remote sensing classification method that integrates cross-scale feature refinement and hybrid state space model (Mamba), mainly used for joint land cover classification of hyperspectral and lidar (LiDAR) data. Background Technology

[0002] With the rapid development of remote sensing observation platform technology, high-quality remote sensing imagery plays a crucial role in urban planning, mineral exploration, and ecological monitoring. Hyperspectral imaging (HSI) contains hundreds of continuous spectral bands, capable of capturing the unique "spectral fingerprints" of ground features, providing rich evidence for fine-grained material identification. However, limited by its imaging mechanism, HSI lacks elevation information and often exhibits poor discrimination in shadowed areas or scenes with significant intraclass spectral variations. In contrast, LiDAR, as an active remote sensing technology, is not limited by lighting conditions and can accurately measure the three-dimensional structural information of the Earth's surface. Combining the spectral properties of HSI with the elevation and geometric features of LiDAR can significantly enhance the robustness of classification models in complex scenes.

[0003] In recent years, deep learning has been widely used in multimodal remote sensing classification. Early methods based on convolutional neural networks (CNNs) performed well in extracting local spectral-spatial features, but were limited by their inherent local receptive fields, making it difficult to effectively model long-range global dependencies. The subsequently introduced Visual Transformer (ViT) solved the global perception problem with its self-attention mechanism, but its computational complexity increases quadratically with the length of the input sequence, leading to huge computational and memory overhead when processing massive high-dimensional remote sensing data. Newly emerging state-space models (such as Mamba) achieve a balance between global modeling and linear computational complexity through selective scanning mechanisms. However, existing Mamba-based methods, when handling joint classification of hyperspectral and lidar data, often focus on single-modality or simple fusion strategies, failing to adequately address key issues such as fine alignment of cross-modal heterogeneous features, deep interaction of multi-scale spatial information, and low reliability of classification decisions in complex scenarios. Summary of the Invention

[0004] Objective: To overcome the problems of weak cross-modal feature correlation, insufficient multi-scale spatial information capture, high computational complexity, and single classification decision path in existing hyperspectral and lidar joint classification methods, this invention proposes a multimodal remote sensing classification method that integrates cross-scale feature refinement and a hybrid state-space model. This method constructs a bi-branch multi-scale adaptive fusion framework, introduces cross-modal global spatial attention (CGSA) and cross-scale feature interaction mechanism (CSMA), and, while ensuring linear computational efficiency, combines Bayesian inference and contrastive learning optimization to achieve efficient, stable, and high-precision multimodal land cover classification.

[0005] Technical solution:

[0006] To achieve efficient and accurate joint classification of hyperspectral and lidar data, and to address the challenges of cross-modal heterogeneous feature alignment, insufficient multi-scale spatial information capture, and low classification decision reliability in complex scenarios in existing multimodal methods, this invention...

[0007] This paper proposes a multimodal remote sensing classification method that integrates cross-scale feature refinement and a hybrid state-space model. The method uses hyperspectral imagery and lidar data as inputs to construct a two-branch, multi-scale adaptive fusion framework. First, a multi-level convolutional neural network is used to extract multi-scale local features, and a cross-modal global spatial attention mechanism is used to achieve initial alignment of heterogeneous features. Then, a cross-scale, cross-modal attention mechanism module is introduced, using a triangular cyclic interaction strategy for fine-grained semantic complementarity and feature refinement. Next, a hybrid state-space model (Mamba) is used to complete long-range dependency modeling while maintaining linear computational complexity. Finally, a two-branch Bayesian classifier combining Monte Carlo random deactivation is constructed, and cross-modal contrastive learning is used to optimize feature distribution, outputting a classification decision that balances high accuracy and high reliability. The specific steps include the following:

[0008] S1: Preprocess the raw hyperspectral image data and lidar data. Gaussian smoothing and the Sobel operator are used to extract edge feature data from the lidar data, and a multi-scale sliding window strategy is adopted to obtain image patch data at three different scales: hyperspectral, lidar, and edge feature data.

[0009] S2: Construct a multi-scale feature extraction module, using a multi-level convolutional neural network to encode image patches of three modalities respectively. In this process, a cross-modal global spatial attention mechanism is introduced to enhance LiDAR features using edge features, and then spatially calibrate hyperspectral features using the enhanced LiDAR features to generate spatially aligned multi-scale feature representations.

[0010] S3: Construct a cross-scale, cross-modal attention module to map features at different scales into sequences of uniform length. A triangular cyclic interaction strategy is employed, using features at one scale as the query vector and features at the other two scales as the key and value vectors, respectively, for attention computation. This achieves semantic complementarity and feature refinement across different scales.

[0011] S4: Construct a hybrid state-space model sequence fusion module, utilizing a selective scanning mechanism to model the long-range dependencies of the refined feature sequences. The continuous system is transformed into a discrete system through discretization parameters, and learnable scale weight parameters are introduced. The processed multi-scale sequence features are then adaptively weighted and fused to obtain a global long-range feature representation.

[0012] S5: Construct a two-branch Bayesian classification module, including a global semantic branch based on a state-space model and a local Bayesian branch based on a convolutional neural network. The local branch introduces a Monte Carlo random deactivation mechanism to quantify uncertainty. Finally, by combining cross-modal contrastive learning loss and classification loss, the prediction results of the two branches are adaptively weighted and fused to obtain the final land cover classification decision.

[0013] Furthermore, the specific feature extraction and fusion process of the cross-modal global spatial attention mechanism in step S2 is divided into two specific steps:

[0014] S21: First, edge features are generated based on LiDAR data. To suppress noise and maintain edge integrity, a combination of Gaussian smoothing and gradient calculation is used. The calculation formula is as follows:

[0015]

[0016] in, The standard deviation is expressed as Gaussian smoothing function, and Representing the horizontal and vertical directions respectively

[0017] Sobel operator gradient operation, For min-max normalization operation, Set the edge filtering threshold function. .

[0018] S22: Perform cascaded dynamic feature fusion. For channels with a number of channels... hyperspectral features LiDAR characteristics and edge features ,pass Convolution maps it to a dimension of The query and key space. In the first stage, the spatial dimension is flattened into a sequence, and the LiDAR features are used as query vectors. Edge features as key vectors Sum value vector Calculate global spatial correlation to generate structurally enhanced lidar features. In the second stage, hyperspectral features are used as query vectors. The enhanced lidar features as a key vector Sum value vector The final fusion features are calculated. The core attention fusion calculation process can be uniformly represented as:

[0019]

[0020] in, This represents the operation of flattening the spatial dimensions of the feature map into a sequence. Scaling factor For the normalized exponential function along the sequence length dimension, For learnable fusion weight parameters, The original image format used to query features is used to form residual connections, ultimately outputting a spatially aligned multi-scale feature representation. , , .

[0021] Furthermore, the specific learning process of the cross-scale, cross-modal attention module in step S3 includes the following two sub-steps:

[0022] S31: The features at the three scales output from step S2 , , Each feature is flattened into a one-dimensional sequence. An independent linear adaptive scaling layer is introduced to force features at different scales to be mapped to a uniform sequence length. and hidden layer dimensions Subsequently, learnable sequence position encoding is introduced. To compensate for the spatial information loss caused by serialization, and to obtain the initial cross-scale sequence through the Dropout layer. , , .

[0023] S32: A triangular cyclic interaction strategy is used for feature updating. (Based on scale 1 features) Taking the update as an example, we set it as the query vector and use the scale 2 features. Set as a key vector, and use scale 3 features Set as a value vector, and project it through a linear projection matrix. , , Attention aggregation is performed after mapping:

[0024] ,

[0025] ,

[0026] ,

[0027]

[0028] Similarly, updating the features at scale 2 uses scale 2 as the query, scale 3 as the key, and scale 1 as the value; the updates at scale 3...

[0029] The update of the feature utilizes scale 3 for querying, scale 1 for keys, and scale 2 for values, thereby achieving deep interaction and flow of information across all scales.

[0030] Furthermore, the specific process of the hybrid state-space model sequence fusion module in step S4 includes the following three sub-steps:

[0031] S41: Introduces a cross-scale guidance mechanism based on element-wise multiplication, utilizing minimum-scale (high-resolution) features. Each guides large-scale features, i.e., execution and .

[0032] S42: Input the above sequence into the hybrid state-space model. Discretize the parameters of the continuous-time state-space model using the zero-order preservation method:

[0033] ,

[0034]

[0035] Introducing a selective scanning mechanism, based on the input sequence Dynamically adjust parameters , With time scale This enables the filtering of irrelevant noise and the selective memorization of global information.

[0036] ,

[0037] ,

[0038]

[0039] in, Indicates a linear projection layer. This is the activation function. After Mamba modeling is completed, the residuals are summed and then flattened back into a two-dimensional spatial image format. Convolution, batch normalization, and ReLU activation function are used to extract local features, which are then flattened into a sequence.

[0040] S43: Introducing learnable multi-scale weight parameters and (Initial value 0.5), the three processed multi-scale sequence features are adaptively weighted and fused to obtain the final global long-range collaborative features. :

[0041]

[0042] Furthermore, the specific process of the two-branch Bayes classification module and loss optimization in step S5 includes the following three sub-steps:

[0043] S51: Construct a global semantic branch and extract Mamba fusion sequence features. The first token (ClassToken) is used as a global context representative, and a global prediction score is generated through layer normalization (LayerNorm) and linear projection. .

[0044] S52: Construct a local Bayesian branch, using the high-resolution local features output from stage S2 as input, sequentially passing them through channel-reduction convolution and global average pooling layers, and then inputting them into a multilayer perceptron containing Monte Carlo Dropout layers. This is executed during the inference phase. After a random forward propagation, the mean prediction is calculated as the local prediction result. And use the prediction variance to quantify decision uncertainty:

[0045]

[0046] S53: Introduce cross-modal contrastive learning loss To optimize feature distribution, hyperspectral features are extracted from the CNN encoder. Features of LiDAR After global pooling, the contrastive loss is calculated to shorten the distance between similar heterogeneous features and widen the distance between features of different classes.

[0047]

[0048] in, Represents cosine similarity. For temperature coefficient, For the positive sample set, For the entire sample set, learnable weights are ultimately used. , The system integrates global and local prediction results and performs end-to-end optimization by combining classification cross-entropy loss and contrast loss.

[0049] Beneficial effects: (1) This invention designs a cross-modal global spatial attention mechanism and uses the flattened QKV paradigm to model global spatial correlation, realizing dynamic filtering and precise alignment of hyperspectral, lidar and edge features, effectively suppressing noise interference in heterogeneous data fusion. (2) This invention introduces a cross-scale cross-modal attention mechanism and adopts a triangular cyclic interaction strategy to break the limitations of traditional independent scales. It enables features at a specific scale to dynamically retrieve supplementary information from other scales, effectively bridging the semantic gap between multi-scale representations and enhancing the ability to capture multi-scale spatial information. (3) This invention applies the Mamba hybrid state-space model to multi-scale sequence fusion, while maintaining linear computational complexity. At the same time, it achieves accurate modeling of global long-distance sequence dependencies, overcomes the bottleneck of huge computational overhead of traditional Transformer models, and meets the high-efficiency processing requirements of complex remote sensing scenarios. (4) This invention constructs a two-branch Bayesian classifier that combines Monte Carlo random deactivation with cross-modal contrastive learning (CMCL) optimization strategy. While narrowing the distance between similar heterogeneous features, it achieves effective quantification of model prediction uncertainty, which greatly improves the decision reliability and robustness of the model in complex land cover classification. Attached Figure Description

[0050] Figure 1 This is a network model diagram for an embodiment of the present invention;

[0051] Figure 2 The diagram shows the cross-modal global spatial attention (CGSA) module, where (a) is a schematic diagram of the multi-scale two-level cascaded feature fusion architecture, and (b) is a schematic diagram of the internal principle of the cross attention computation unit.

[0052] Figure 3 A diagram of the Cross-Scale Cross-Modal Attention (CSMA) module;

[0053] Figure 4 Here is a diagram of the Mamba state-space model structure.

[0054] Figure 5 Diagram of the Bayesian inference classifier structure;

[0055] Figure 6 The images show the classification results of the model on Houston 2013, where (a) is the result of the real ground cover, (b) is the classification result of CCRNet, (c) is the classification result of FusAtNet model, (d) is the classification result of MFT model, (e) is the classification result of HLMamba model, and (f) is the classification result of MAFMamba model of this invention. Detailed Implementation

[0056] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0057] This invention discloses a multimodal remote sensing classification network that integrates cross-scale feature refinement and a hybrid state-space model. The network mainly includes multimodal data preprocessing; constructing a classification network comprising a multi-scale feature extraction module, a cross-scale cross-modal attention module, a hybrid state-space model sequence fusion module, and a two-branch Bayesian classification module. This approach reduces computational complexity while maintaining global long-range dependency modeling capabilities and high-precision classification, meeting the high-reliability classification requirements of complex terrain scenarios.

[0058] Based on the above method, this embodiment applies the method to the joint classification of hyperspectral and lidar systems, such as... Figure 1 As shown, the specific process is as follows:

[0059] S1: Preprocessing and multi-scale feature construction of the raw hyperspectral image data and lidar data. Specifically, this step mainly includes the following three sub-steps:

[0060] S11: Acquire hyperspectral image (HSI) and single-channel lidar (LiDAR) elevation data of the same area, and normalize the raw data;

[0061] S12: Edge contour feature data is extracted from LiDAR data using Gaussian smoothing and the Sobel operator. The extraction calculation formula is as follows:

[0062]

[0063] in, It is a Gaussian smoothing function. and These represent the gradient calculations performed by the horizontal and vertical Sobel kernels, respectively. Represents normalization operation, For threshold function;

[0064] S13: Centered on the current pixel to be classified, use... A multi-scale sliding window strategy, supplemented by symmetrical filling, was used to acquire image patch data at three different scales: hyperspectral, lidar, and edge features.

[0065] S2: Construct a multi-scale feature extraction module, utilizing a multi-level convolutional neural network to extract features from each modality and perform cross-modal global spatial attention (CGSA) fusion. For example... Figure 2 As shown, specifically, this step mainly includes the following three sub-steps:

[0066] S21: A structurally symmetric two-stage convolutional network is used to extract deep semantic features from hyperspectral data and elevation structural features from lidar data, respectively. The batch normalization layer is then removed, and... Convolution is used to extract edge contour features to prevent blurring of the contours;

[0067] S22: The first stage of cascaded dynamic feature fusion is performed, flattening the spatial dimension into a sequence and using LiDAR features as the query vector. Edge features as key vectors Sum value vector Calculate global spatial correlation to generate structurally enhanced lidar features. ;

[0068] S23: Perform the second-stage fusion, using hyperspectral features as the primary query vector. The laser radar with the enhanced structure

[0069] Characteristics as a key vector Sum value vector The final fused features are calculated, and the spatially aligned multi-scale feature representation is output. The core attention calculation process is uniformly represented as follows:

[0070]

[0071] in, This represents the operation of flattening the spatial dimensions of the feature map into a sequence. This is the scaling factor.

[0072] S3: Construct a cross-scale, cross-modal attention (CSMA) module to achieve semantic complementarity and feature refinement across different scales. For example... Figure 3 As shown, specifically, this step mainly includes the following two sub-steps:

[0073] S31: Map the features at the three different scales output from step S2 to a uniform sequence length through a linear embedding layer. and hidden layer dimensions Furthermore, learnable positional encoding is introduced to supplement the spatial information lost during serialization;

[0074] S32: A triangular cyclic interaction strategy is used for feature updating. (Based on scale 1 features) Taking the update as an example, let's set it as a query vector. Set the scale 2 features as key vectors The scale 3 features are set as value vectors. Information is aggregated through attention mechanisms:

[0075]

[0076] Similarly, updating features at scale 2 uses scale 2 as the query, scale 3 as the key, and scale 1 as the value; updating features at scale 3 uses scale 3 as the query, scale 1 as the key, and scale 2 as the value, thus achieving information interaction across all scales.

[0077] S4: Construct a hybrid state-space model (Mamba) sequence fusion module to perform long-range dependency modeling on the refined feature sequences. For example... Figure 4 As shown, specifically, this step mainly includes the following three sub-steps:

[0078] S41: Discretize the parameters of the continuous-time state-space model using the zero-order preservation method to obtain discretized parameters suitable for recursion in deep networks:

[0079]

[0080]

[0081] S42: Introduces a selective scanning mechanism (S6) to dynamically adjust parameters based on the input sequence. , With time scale This enables the filtering of irrelevant noise and the selective memorization of global information;

[0082] S43: Introduce learnable scale weight parameters and The three multi-scale sequence features processed by the state-space model are adaptively weighted and fused to obtain a global long-range feature representation:

[0083]

[0084] S5: Construct a two-branch Bayesian classification module, quantify uncertainty, and output the final classification decision. For example... Figure 5 As shown, specifically, this step mainly includes the following three sub-steps:

[0085] S51: Construct a global semantic branch and extract the first token (ClassToken) from the fusion output sequence of the state-space model.

[0086] As a global context representative, a global prediction score is generated through linear projection. ;

[0087] S52: Construct a local Bayesian branch, taking the highest resolution (scale 1) fused features as input, sequentially passing them through convolutional layers and global average pooling layers, and then inputting them into a multilayer perceptron containing a random dropout layer. During the inference phase, perform T random forward propagations, calculating the prediction mean as the local prediction result. And use variance to quantify uncertainty:

[0088]

[0089] S53: Introduce cross-modal contrastive learning loss Optimize feature distribution and utilize learnable weights. , Adaptive weighted fusion of global and local prediction results: Finally, the classification cross-entropy loss and contrastive learning loss are combined for end-to-end optimization to output the land cover classification decision.

[0090] Analysis of Simulation Experiment Results

[0091] 1. Experimental Images

[0092] This invention will test the proposed fusion of cross-scale feature refinement and hybrid state space model (MAFMamba) on the Houston (Houston2013) benchmark multimodal remote sensing dataset to verify the effectiveness and reliability of the model method of this invention.

[0093] The Houston2013 dataset was collected by a Compact Airborne Hyperspectral Imager (CASI) over the University of Houston campus and surrounding urban areas. The images have a spatial dimension of 345 pixels * 1905 pixels and a spatial resolution of 2.5 m. The hyperspectral data contains 144 spectral bands, ranging from 0.38 to 1.35 μm in wavelength. Additionally, the dataset includes lidar-derived digital surface model (DSM) elevation data for the same area. Only 15,029 pixels contain feature labels, representing 15 complex feature classes, such as healthy grasslands, trees, residential areas, and roads.

[0094] 2. Experimental methods and related parameter settings

[0095] The experimental computer was configured with an Intel Xeon E5-2680 v4 CPU, an NVIDIA GeForce RTX 4090 GPU, an Ubuntu operating system, Python as the programming language, and PyTorch as the deep learning model framework.

[0096] For the experimental dataset, the learning rate was set to 3e-4, the batch size was 64, and the patch sizes for multi-scale inputs were set to 8×8, 16×16, and 24×24, respectively.

[0097] To objectively evaluate model performance, three commonly used metrics were employed: overall accuracy (OA), average accuracy (AA), and the Kappa coefficient. All experimental results are the average of five independent runs.

[0098] In the partitioning of the training and test sets, a fixed number of samples were selected for each class on the Houston2013 dataset as training samples, and the remaining samples were used as test samples. The specific number of samples selected for each class is shown in Table 1. In the experiment, the batch size was set to 64, the backpropagation algorithm used was Adam, the initial learning rate was 0.0003, and the number of iterations was set to 200.

[0099] Table 1 - Ground truth map and land cover information categories for the Houston2013 dataset, and the number of samples used for training and testing for each class.

[0100] Healthy Grass 198 1053 Stressed Grass 190 1064 Synthetic Grass 192 505 Trees 188 1056 Soil 186 1056 Water 182 143 Residential 196 1072 Commercial 191 1053 Road 193 1059 Highway 191 1036 Railway 181 1054 Parking Lot1 192 1041 Parking Lot2 184 285 Tennis Court 181 247 Running Track 187 473 total 2832 12197

[0101] 3. Comparison of Experimental Results

[0102] Table 2 - Houston2013 Dataset Test Results

[0103] Healthy Grass 93.18 98.84 98.73 82.53 82.81 Stressed Grass 85.17 83.31 88.43 100.00 100.00 Synthetic Grass 89.36 91.68 84.81 94.85 98.61 Trees 96.53 95.73 95.80 99.90 100.00 Soil 96.08 97.49 86.48 100.00 100.00 Water 74.07 82.55 93.73 95.80 100.00 Residential 84.34 80.30 73.76 90.58 87.31 Commercial 84.93 77.47 88.38 94.40 94.02 Road 89.03 78.33 87.66 99.06 93.30 Highway 93.86 83.22 71.02 88.80 96.04 Railway 96.03 83.86 70.03 89.28 95.07 Parking Lot1 82.05 75.97 92.59 78.48 89.24 Parking Lot2 89.22 67.94 77.63 91.93 89.47 Tennis Court 70.26 76.57 66.72 100.00 98.79 Running Track 89.22 91.98 98.45 97.88 100.00 OA 89.06 84.78 84.54 92.84 94.29 AA 88.90 83.60 86.90 93.57 94.98 Kappa 88.13 83.47 83.31 92.23 93.81

Claims

1. A multimodal remote sensing classification method integrating cross-scale feature refinement and a hybrid state-space model, characterized in that, Includes the following steps: S1: Preprocess the raw hyperspectral image data and lidar data. Gaussian smoothing and the Sobel operator are used to extract edge feature data from the lidar data, and a multi-scale sliding window strategy is adopted to obtain image patch data at three different scales: hyperspectral, lidar, and edge feature data. S2: Construct a multi-scale feature extraction module, using a multi-level convolutional neural network to encode image patches of three modalities respectively. In this process, a cross-modal global spatial attention mechanism is introduced to enhance the LiDAR features using edge features, and then the enhanced LiDAR features are used to spatially calibrate the hyperspectral features to generate spatially aligned multi-scale feature representations. S3: Construct a cross-scale, cross-modal attention module to map features at different scales into sequences of uniform length. A triangular cyclic interaction strategy is employed, using features at one scale as the query vector and features at the other two scales as the key and value vectors, respectively, for attention computation. This achieves semantic complementarity and feature refinement across different scales. S4: Construct a hybrid state-space model sequence fusion module, utilizing a selective scanning mechanism to model the long-range dependencies of the refined feature sequences. The continuous system is transformed into a discrete system through discretization parameters, and learnable scale weight parameters are introduced. The processed multi-scale sequence features are then adaptively weighted and fused to obtain a global long-range feature representation. S5: Construct a two-branch Bayesian classification module, including a global semantic branch based on a state-space model and a local Bayesian branch based on a convolutional neural network. The local branch introduces a Monte Carlo random deactivation mechanism to quantify uncertainty. Finally, by combining cross-modal contrastive learning loss and classification loss, the prediction results of the two branches are adaptively weighted and fused to obtain the final land cover classification decision.

2. The multimodal remote sensing classification method according to claim 1, which integrates cross-scale feature refinement and a hybrid state-space model, is characterized in that... The specific feature extraction and fusion process of the cross-modal global spatial attention mechanism in step S2 includes: S21: First, edge features are generated based on LiDAR data. The calculation formula is as follows: in, Represents the Gaussian smoothing function. and These represent gradient calculations in the horizontal and vertical directions, respectively. For normalization operations, This is a threshold function. S22: Perform cascaded dynamic feature fusion. In the first stage, LiDAR features are used as the query vector. Edge features as key vectors Sum value vector Calculate global spatial correlation to generate structurally enhanced lidar features. In the second stage, hyperspectral features are used as query vectors. The enhanced lidar features as a key vector Sum value vector The final fusion features are calculated. The core attention calculation process can be uniformly represented as: in, This represents the operation of flattening the spatial dimensions of the feature map into a sequence. Scaling factor It is a normalized exponential function.

3. The multimodal remote sensing classification method according to claim 1, which integrates cross-scale feature refinement and a hybrid state-space model, is characterized in that... The specific learning process of the cross-scale, cross-modal attention module in step S3 includes: S31: Input features at three different scales , , Mapped to a uniform sequence length through linear embedding layers respectively and hidden layer dimensions And based on this, position coding is introduced. S32: A triangular cyclic interaction strategy is used for feature updating. (Based on scale 1 features) Taking the update as an example, let's set it as a query vector. Scale 2 features Set as key vector Scale 3 features Set as a value vector Information is aggregated through attention mechanisms: in, , , It is a linear projection matrix. Similarly, the update of features at scale 2 uses scale 2 as the query, scale 3 as the key, and scale 1 as the value; the update of features at scale 3 uses scale 3 as the query, scale 1 as the key, and scale 2 as the value, thereby realizing the information flow across all scales.

4. The multimodal remote sensing classification method according to claim 1, which integrates cross-scale feature refinement and a hybrid state-space model, is characterized in that... The specific process of the hybrid state-space model sequence fusion module in step S4 includes: S41: The parameters of the continuous-time state-space model are discretized using the zero-order preservation method for recursive computation of deep learning models. The specific discretization formula is as follows: in, For time scale parameters, Here is the state transition matrix. For the input projection matrix, It is an identity matrix. S42: Introduces a selective scanning mechanism based on the input sequence. Dynamically adjust parameters , , This enables the filtering of irrelevant noise and the selective memorization of global information. in, Indicates a linear projection layer. This is the activation function. S43: The three scale features after processing by the state-space model , , Learnable weight parameters and Adaptive fusion is performed to obtain the final multi-scale collaborative features. :

5. The multimodal remote sensing classification method according to claim 1, which integrates cross-scale feature refinement and a hybrid state-space model, is characterized in that... The specific process of step S5, the bi-branch Bayesian classification module and loss optimization, includes: S51: Construct a global semantic branch, extract the first label of the output sequence of the state-space model as the global context representative, and generate a global prediction score through linear projection. . S52: Construct a local Bayesian branch, using the lowest-scale high-resolution features as input, sequentially passing them through convolutional layers and global average pooling layers, and then inputting them into a multilayer perceptron containing random deactivation layers. This is executed during the inference phase. After a random forward propagation, the mean prediction is calculated as the local prediction result. And use the prediction variance to quantify uncertainty: S53: Introduce cross-modal contrastive learning loss To optimize feature distribution, the distance between similar heterogeneous features is reduced, while the distance between features of different classes is increased. in, Represents cosine similarity. For temperature coefficient, For the positive sample set, For the entire sample set, learnable weights are ultimately used. , The system integrates global and local prediction results and performs end-to-end optimization by combining classification loss and contrastive loss.

Images