Hyperspectral and lidar data combined ground feature classification method, device and medium

Abstract

The application belongs to the technical field of remote sensing data processing, and discloses a hyperspectral and laser radar data joint ground feature classification method, equipment and medium, the method comprises the following steps: acquiring hyperspectral image data and laser radar elevation data; feature extraction is carried out by adopting a three-branch parallel structure, wherein the hyperspectral image branch extracts global spectral spatial features by three-dimensional convolution and bidirectional scanning along the spectral dimension, the laser radar branch extracts elevation structure features by multi-level convolution, and the multi-modal branch extracts multi-modal fusion features by early fusion and a multi-scale feature enhancement module; a multi-head bidirectional cross attention module is used to perform deep bidirectional interaction and fusion on the features extracted by the three branches, thereby generating a final fusion feature vector; finally, the final fusion feature vector is input into a classifier to obtain a classification result. The method realizes deep interaction between modes and adaptive perception of multi-scale ground features, and significantly improves the accuracy of joint classification.

Description

Technical Field

[0001] This invention relates to the field of remote sensing data processing technology, and in particular to a method, device and medium for land cover classification based on a multi-scale cross-attention fusion network that combines hyperspectral and lidar data. Background Technology

[0002] Remote sensing technology, through collaborative observation using multiple platforms and sensors, provides increasingly rich multi-source data for Earth observation. Among these, hyperspectral imaging (HSI) and lidar (LiDAR) data are two highly complementary data sources. HSI provides detailed spectral information about ground features but has limitations in spatial structure description; while LiDAR data provides accurate three-dimensional elevation information, effectively compensating for the spatial dimensionality deficiencies of HSI. Therefore, joint classification of HSI and LiDAR data can significantly improve the accuracy of land cover classification, and has broad application prospects in fields such as urban planning, environmental monitoring, and resource exploration.

[0003] In existing technologies, deep learning-based methods, particularly convolutional neural networks (CNNs), Transformers, and state-space models (such as Mamba), have been widely applied to joint classification tasks for HSI and LiDAR. CNN-based methods extract and fuse features by designing complex network structures or introducing attention mechanisms, but their inherent local receptive fields limit their ability to model long-range dependencies. While Transformer-based models can capture global dependencies, they suffer from high computational complexity, and their fixed labeling strategies struggle to adapt to variations in ground-scale features. The emerging Mamba model, although showing some efficiency improvements, still requires further refinement in handling complex spatial dependencies.

[0004] In summary, existing HSI-LiDAR joint classification methods generally face the following technical problems: (1) Existing model architectures have limitations. For example, CNNs are difficult to model long-distance dependencies, while Transformer and Mamba are not flexible enough when dealing with multi-scale features and cannot simultaneously take into account local details and global context.

[0005] (2) Insufficient cross-modal feature interaction. Many methods only perform simple splicing or addition fusion in the later stage of feature extraction, failing to achieve deep information interaction and collaborative learning between modalities, resulting in insufficient utilization of complementary information.

[0006] (3) Poor adaptability to multi-scale changes of ground objects in complex scenes. Existing methods are difficult to dynamically integrate features of different scales, resulting in a decline in classification performance when facing ground objects with significant size differences.

[0007] Therefore, there is an urgent need to propose a new technical solution that can effectively integrate the advantages of different models, realize deep bidirectional interaction between modalities, and have powerful multi-scale feature perception capabilities, so as to systematically solve the above-mentioned technical bottlenecks in the current HSI-LiDAR joint classification method. Summary of the Invention

[0008] The main objective of this invention is to provide a method, device, and medium for land cover classification based on a multi-scale cross-attention fusion network that combines hyperspectral and lidar data. This aims to solve problems in existing technologies, such as insufficient multimodal feature interaction, poor multi-scale adaptability, and inherent limitations of the model architecture during land cover classification.

[0009] In a first aspect, the present invention provides a method for land cover classification that combines hyperspectral and lidar data, comprising the following steps: Acquire hyperspectral image data and lidar elevation data of the ground features to be processed; A three-branch parallel structure is used to extract features from the hyperspectral image data and the lidar elevation data. The three-branch parallel structure includes a hyperspectral image branch, a lidar branch, and a multimodal branch. In the hyperspectral image branch, the hyperspectral image data is subjected to three-dimensional convolution and bidirectional scanning along the spectral dimension to extract global spectral spatial features; In the lidar branch, the lidar elevation data undergoes multi-level convolution processing to extract elevation structure features; In the multimodal branch, the hyperspectral image data and the lidar elevation data are fused in the early stage and processed by the multi-scale feature enhancement module to extract multimodal fusion features. The multi-scale feature enhancement module achieves multi-scale contextual information capture through parallel dilated convolutions with different dilation rates. A multi-head bidirectional cross-attention module is used to perform deep fusion processing on the global spectral spatial features, the elevation structure features, and the multi-scale fusion features. The deep fusion processing includes performing bidirectional cross-attention calculation on any two features and aggregating all interaction results to generate the final fused feature vector. The final fused feature vector is input into the classifier, which outputs the land cover classification results corresponding to the hyperspectral image data and the lidar elevation data.

[0010] As an optional implementation of the first aspect of this application, the step of extracting global spectral spatial features specifically includes: converting the input hyperspectral image data into a three-dimensional feature cube. Feature extraction is performed using three-dimensional depthwise separable convolutional blocks. The calculation process of the three-dimensional depthwise separable convolutional blocks is as follows: ; ;in, For depthwise convolution output, For pointwise convolution output, DWCov3×3×3 is a 3D depthwise convolution, Conv1×1×1 is a 1×1×1 pointwise convolution, BN3D is 3D batch normalization, and ζ(·) is the GELU activation function; for the pointwise convolution output Pooling and activation processes are performed to obtain post-processed features. The post-processing features are analyzed using a 3D scanning module. The 3D scanning module performs processing on the post-processed features. After serialization along the spectral dimension, a bidirectional scan is performed, both forward and backward, and the results of the bidirectional scan are averaged to obtain the spectral scan features. The spectral scanning features are layer-normalized and compared with the post-processed features. Perform residual connections to obtain the global spectral spatial features. The calculation process is as follows: .

[0011] As an optional implementation of the first aspect of this application, the step of extracting elevation structure features specifically includes: using the lidar elevation data as initial input. Layered feature extraction is performed through N consecutive convolutional blocks, where the output feature of the i-th convolutional block is... The calculation process is as follows: Where Conv3×3 is a 3×3 two-dimensional convolution, BN is two-dimensional batch normalization, and ξ(·) is the ReLU activation function. The output feature of the (i-1)th convolutional block is used; the output of the last convolutional block is used as the elevation structure feature. The elevation structural features The dimension and the global spectral spatial features The dimensions remain consistent.

[0012] As an optional implementation of the first aspect of this application, the specific process of the early fusion in the step of extracting multi-scale fusion features includes: processing the hyperspectral image data... With the lidar elevation data The data is stitched along the channel dimension; the stitched data is then projected and nonlinearly transformed through a 1×1 two-dimensional convolutional layer; the transformed features are then compared with the original hyperspectral image data. Perform residual connections to obtain early fusion features. The calculation process is as follows: ; where [·,·] represents the concatenation operation along the channel dimension, Conv1×1 is a 1×1 two-dimensional convolution, and ξ(·) is the ReLU activation function.

[0013] As an optional implementation of the first aspect of this application, after obtaining the aforementioned early fusion features... Subsequently, the step of processing the early fusion features through the multi-scale feature enhancement module to extract multimodal fusion features specifically includes: processing the early fusion features... Intermediate features are obtained by sequentially processing convolutional blocks and two-dimensional depthwise separable convolutional blocks. ; the intermediate features The input is fed into the multi-scale feature enhancement module, which captures multi-scale contextual information through N parallel dilated convolutional branches with different dilation rates k. The output of the k-th scale branch... The calculation process is as follows: Where Conv3×3,dilation=k represents a 3×3 convolution with a dilation rate of k; the outputs of all scale branches are concatenated to obtain multi-scale fused features. The calculation process is as follows: ; for the multi-scale fusion features The multimodal fusion features are obtained by performing convolutional enhancement and channel attention weighting. The calculation process for the channel attention weighting is as follows: ; ;in, The features are enhanced by convolution. GAP is global average pooling, σ is the Sigmoid activation function, and ζ is the GELU activation function. This represents element-wise multiplication along the channel dimension.

[0014] As an optional implementation of the first aspect of this application, the deep fusion processing includes the step of performing bidirectional cross-attention calculation on any two features, including the global spectral spatial features. and the aforementioned elevation structural features The step of performing bidirectional cross-attention calculation specifically includes: processing the input global spectral spatial features... and the aforementioned elevation structural features Perform layer normalization; project each normalized feature through a linear projection layer to generate a query vector Q, a key vector K, and a value vector V; perform bidirectional cross-attention calculation, where the first-way cross-attention calculation uses... query vector and key vector Sum value vector To enable interaction, the second-way cross-attention calculation is used. query vector and key vector Sum value vector Perform interaction; combine the output of the first-way cross-attention calculation with its corresponding query vector. Perform a residual connection, and combine the output of the second-direction cross-attention calculation with its corresponding query vector. Perform residual connections to preserve the original information of each component, resulting in a bidirectional interactive output representation. .

[0015] As an optional implementation of the first aspect of this application, the deep fusion processing step further includes: sequentially processing global spectral spatial features Multi-scale fusion features Global spectral spatial characteristics With elevation structural features Multi-scale fusion features With elevation structural features These three feature pairs undergo bidirectional cross-attention calculation; the results of the bidirectional cross-attention calculations for all three feature pairs are summed to obtain the aggregated features; the aggregated features are then subjected to global average pooling to convert them into a one-dimensional vector, resulting in the final fused feature vector. .

[0016] As an optional implementation of the first aspect of this application, the step of inputting the final fused feature vector into the classifier specifically includes: inputting the final fused feature vector... The input is fed into a classifier consisting of a dropout layer and a linear layer; the classifier outputs the class prediction probability for each pixel through a linear transformation; during the model training phase, the cross-entropy loss function is used to calculate the loss between the predicted probability and the true label, and the model parameters are updated through backpropagation until the model converges.

[0017] Secondly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0018] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By adopting an innovative three-branch parallel architecture, modality-specific feature extraction (hyperspectral image branch and lidar branch) is decoupled from multimodal fusion feature extraction (multimodal branch). This not only preserves the unique information of each modality, but also provides a structured foundation for subsequent deep interaction, effectively avoiding the feature confusion caused by early fusion or the insufficient interaction caused by late fusion in traditional dual-branch networks.

[0020] 2. A multi-scale feature enhancement (MSFE) module was designed, which utilizes parallel multi-scale dilated convolution and adaptive channel attention mechanism to enable the model to dynamically perceive and fuse spatial spectral features at different scales, significantly improving the model's adaptability and representation ability to ground objects with varying sizes in complex scenes.

[0021] 3. A multi-head bidirectional cross-attention (MBCA) module is proposed. By performing pairwise, bidirectional deep interaction on the features output by the three branches and combining it with a triple residual connection design, full information exchange and semantic alignment between modalities are achieved. The complementarity of hyperspectral imaging (HSI) and lidar (LiDAR) is effectively utilized, thereby improving the accuracy and robustness of the final classification.

[0022] 4. An end-to-end, efficient joint classification framework was constructed, which integrates the local perception capability of CNNs and the long-range dependency modeling advantages of Mamba-like models. Its superiority was verified on multiple public datasets, providing an effective technical solution for high-precision and robust multimodal remote sensing data classification. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall structure of the Multi-Scale Cross-Attention Fusion Network (MCAFNet) for joint classification of HSI and LiDAR data provided in this embodiment of the invention; Figure 2 This is a schematic diagram of (a) the spectral scanning path and (b) the early fusion module provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the multi-scale feature enhancement (MSFE) module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the Multi-Head Bidirectional Cross-Attention (MBCA) module provided in an embodiment of the present invention; Figure 5 This is a visualization of the classification results of the method of this invention on the Houston 2013 dataset; Figure 6 This is a visualization of the classification results of the method of this invention on the Augsburg dataset; Figure 7 This is a visualization of the classification results of the method of this invention on the Trento dataset; Figure 8 This is a visualization of the classification results of the method of this invention on the MUUFL dataset; Figure 9 This is a graph analyzing the impact of different feature dimensions on model performance; Figure 10 This is a graph analyzing the impact of different image patch sizes on model performance. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0026] This paper proposes a multi-scale cross-attention fusion network (MCAFNet) for joint classification of HSI and LiDAR data. Figure 1 The overall framework shown adopts a three-branch encoder architecture. It extracts global spectral spatial features from HSI data and elevation structure features from LiDAR data, and then performs shallow fusion and multi-scale feature enhancement (MSFE) through a multimodal branch. Finally, a designed multi-head bidirectional cross-attention (MBCA) module realizes deep bidirectional feature interaction between the three branches, making full use of complementary information between modalities. The design and principle of each core component will be described in detail below.

[0027] A. Spatial-spectral feature extraction Given an input HSI image patch and LiDAR elevation data blocks The model extracts features through three parallel branches (hyperspectral image branch, LiDAR branch, and multimodal branch). Each branch outputs a feature map of the same dimension, laying the foundation for subsequent deep fusion.

[0028] (1) Hyperspectral Image (HSI) Branch: Global Spectral-Spatial Feature Extraction The HSI branch aims to effectively capture the spectral-spatial dependencies of hyperspectral data through 3D convolution and selective state-space scanning mechanisms. This branch first expands the input hyperspectral data... The depth dimension is then converted into a 3D feature cube. Subsequently, efficient feature extraction is performed using 3D depthwise separable convolutional blocks, primarily through cascaded depthwise separable convolutions and pointwise 3D convolutions: ; ; in, For depthwise convolution output, For pointwise convolution output, DWCov3×3×3 is a 3D depthwise convolution, Conv1×1×1 is a 1×1×1 pointwise convolution, BN3D is a 3D batch normalization, and ζ(·) is the GELU activation function.

[0029] To enhance feature representation and reduce computational complexity, pooling and activation processes are performed to obtain post-processed features: ; in This indicates max pooling. The core of this branch lies in introducing a scanning mechanism inspired by Mamba's selective state-space model, appropriately simplified for our task. To accommodate hyperspectral data, we specifically tailor the scanning process for the spectral dimension, unlike Mamba's general sequence modeling paradigm. We also simplify the state-space update logic to a bidirectional linear transformation with shared parameters, avoiding the complex structured state-space computations found in the original Mamba.

[0030] The 3D-Scan module models long-range dependencies using a bidirectional scanning strategy. The processing flow is as follows: First, the input features are serialized along the spectral dimension to generate a sequence representation. Then, a bidirectional scanning process is performed: the forward scan processes the sequence from front to back along the spectral dimension, integrating the current state with the previous hidden state; the reverse scan processes the sequence in the opposite direction. The specific scan path is as follows... Figure 2 As shown in (a) above, the two scan processes share parameter matrices A and B, which are used to process the current input and the previous state, respectively. The scan output is controlled by a sigmoid gating mechanism, and the bidirectional scan results are finally averaged to form an enhanced feature representation.

[0031] Enhanced feature representation obtains spectral scan features through a simplified state-space scan along the spectral dimension. Finally, output features are obtained through layer normalization and residual connections to ensure training stability. ; This branch outputs global spectral spatial features. It effectively integrates the spectral-spatial structure information of HSI data and enhances long-range dependency modeling through a selective scanning mechanism, among which the feature dimensions It is defined as an adjustable hyperparameter.

[0032] (2) LiDAR Branch: Elevation Feature Extraction The LiDAR branch focuses on extracting multi-level elevation structure features from digital surface model (DSM) data. Given that LiDAR data typically contains rich spatial information but relatively simple spectral features compared to HSI, this branch employs a streamlined convolutional architecture to capture hierarchical elevation patterns.

[0033] The processing begins with LiDAR data and feature extraction is performed through N consecutive convolutional blocks. The output of the i-th convolutional block is given by the following mathematical formula: ; Where, for the initial condition i=0, .

[0034] This branch ultimately outputs features such as elevation structure characteristics. Its dimensions and Careful alignment is required for seamless integration in subsequent fusion modules. The extracted multi-level elevation structure features provide crucial geometric information for the core MBCA module.

[0035] (3) Multimodal branch: Multiscale fusion feature extraction The multimodal branch is one of the core components of our network. Its purpose is to fuse HSI and LiDAR data in the early stages and capture target features through multi-scale processing.

[0036] In the early fusion modules, such as Figure 2 As shown in (b), the HSI and LiDAR input data are first concatenated along the channel dimension to create a unified representation. Then, the combined features are projected and nonlinearly transformed through a 1×1 2D convolutional layer. Finally, early fused features are obtained through residual connections with the original HSI input. This is to preserve crucial spectral information. The early fusion process can be represented mathematically as follows: ; Where [˙,˙] represents the splicing operation along the channel dimension.

[0037] Output characteristics of early fusion modules Although it integrates multimodal information, it is still a relatively shallow representation. Therefore, it is first refined using a convolutional block with a 3×3 convolution followed by batch normalization and ReLU activation to enhance its representational power. The refined features... The data is then fed into a 2D depthwise separable convolutional block (DSCB-2D) for further efficient feature extraction. Intermediate features are obtained through depthwise separable convolution and pointwise convolution. As shown below: ; ; However, ground features in remotely sensed scenes exhibit significant scale variations, ranging from broad, uniform areas to small, isolated objects coexisting in the same image. To enhance the model's adaptability to such multi-scale features, we designed an MSFE module. Figure 3 As shown, the MSFE module captures multi-scale contextual information through N parallel dilated convolutions. Let... This represents the output of the k-th scale branch; the process can be expressed mathematically as follows: ; Where Conv3×3,dilation=k represents a 3×3 convolution with a dilation rate of k, and the number of parallel branches N (corresponding to...) In our implementation, the index k is set to 3, and the hole rate ranges from 1 to 3.

[0038] The outputs of all scale branches are concatenated to obtain the multi-scale fused features. The calculation process is as follows: ; This configuration effectively captures multi-scale spatial context. The motivation behind this design is the need to balance computational efficiency with the ability to represent objects of different scales commonly found in remote sensing scenes.

[0039] To address the issues of information redundancy in multi-scale fusion features and the varying contributions of different channels to classification, we first enhance the feature representation using convolutional blocks to obtain the feature map. Subsequently, a channel attention mechanism was introduced. This mechanism adaptively recalibrates the feature responses and highlights the most discriminative channels. The corresponding mathematical formula is as follows: ; Where GAP represents global average pooling, and σ is the sigmoid activation function. This represents element-wise multiplication along the channel dimension. This branch outputs multimodal fusion features. .

[0040] B. Deep Integration and Classification (1) Multi-head bidirectional cross-attention module To establish deep bidirectional information interaction among the three feature branches (HSI, LiDAR, and multimodal), a multi-head bidirectional cross-attention (MBCA) module was designed. This module facilitates semantic alignment and bidirectional information exchange between features from different branches, fully utilizing complementary information between modalities to enhance joint classification performance. Figure 4 As shown, with features and For example: First, the two input features are normalized through layers to stabilize training and improve feature consistency. Then, each feature is projected onto a query (Q), key (K), and value (V) vector through a linear projection layer, preparing for subsequent attention computation. In the bidirectional cross-attention computation, the Q vector of each branch interacts with the K and V vectors of the other branch, thereby retrieving and aggregating relevant information from the features of the other branch. Specifically, the Q of branch A is attention-processed with the KV pair of branch B to obtain information beneficial to A; similarly, branch B extracts useful information from branch A. The output of each attention computation is added to its original Q through a residual connection to retain key information. Finally, the output representation after bidirectional interaction is obtained. .

[0041] Finally, the three feature pairs ( , , The above bidirectional interaction is performed, and the results of all interactions are summed. To facilitate prediction by the subsequent classification head, the summed features are processed by global average pooling (GAP) to form a rich final fused feature vector. The MBCA module procedure is represented by the following mathematical expression: .

[0042] (2) Classifier Final fusion features The data is now fed as 1D vectors into a classifier consisting of dropout and linear layers to predict the class label for each pixel. This design helps prevent overfitting and improves the model's generalization ability. The model is trained to minimize the cross-entropy loss between the prediction and the true label. Through end-to-end training, MCAFNet can automatically learn how to effectively fuse HSI and LiDAR data to achieve optimal classification performance.

[0043] Experiments and Analysis This section provides a comprehensive overview of the datasets used, evaluation metrics, and experimental setup. Furthermore, comparative experiments and ablation studies were conducted to demonstrate the effectiveness and superiority of the proposed MCAFNet.

[0044] A. Dataset Description To verify the effectiveness of the proposed method, experiments were conducted on four publicly available multi-sensor remote sensing image classification datasets: Houston2013, Augsburg, Trento, and MUUFL. These datasets contain both HSI and LiDAR data, covering various land cover scenarios such as urban and agricultural areas, ensuring fair and publicly comparable data.

[0045] (1) Houston2013 dataset: The Houston 2013 dataset was acquired using a CASI-1500 sensor over the University of Houston area. The dataset has a spatial size of 349 × 1505 pixels, provides 144 spectral bands, and covers a wavelength range of 0.38. The image size is 1.05 μm, with a spatial resolution of 2.5 meters. This dataset also includes a LiDAR-derived digital surface model at the same resolution. The dataset contains 15 classification categories.

[0046] (2) Augsburg dataset: The Augsburg dataset was collected in the city of Augsburg, Germany, and has a spatial dimension of 332 × 485 pixels. This dataset contains hyperspectral images with 180 spectral bands and a wavelength range of 0.44. The dataset has a resolution of 2.5 μm and a spatial resolution of 30 meters, and also includes LiDAR elevation data. It contains ground truth data for seven categories.

[0047] (3) Trento dataset: The Trento dataset represents an agricultural area in Italy and was captured by the AISA Eagle sensor. The dataset has a spatial size of 600×166 pixels, provides 63 spectral bands, and a spatial resolution of 1 meter. It also includes a registered 1-meter resolution LiDAR digital surface model. The dataset contains six distinct land cover categories.

[0048] (4) MUUFL dataset: The MUUFL dataset was acquired over the University of Southern Mississippi's Gulfport campus. The dataset has a spatial dimension of 325 × 220 pixels and contains 64 bands of hyperspectral data, covering visible and near-infrared wavelengths, with a spatial resolution of 0.54 × 1.0 meters, accompanied by LiDAR data at a resolution of 0.60 × 0.78 meters. The dataset includes 11 land cover categories.

[0049] B. Experimental Setup All experiments were conducted on a computer equipped with an Intel Core i5-14600KF CPU, an NVIDIA GeForce RTX4060Ti GPU, and Ubuntu 20.04. Implementation was based on PyTorch. The Adam optimizer was used with a learning rate decay factor of 0.05. Based on empirical validation, the batch size was set to 64, the initial learning rate to 0.0005, and the model was trained for 100 epochs. The cross-entropy loss function was employed. Model performance was evaluated using four metrics: overall accuracy (OA), average accuracy (AA), Kappa coefficient (κ), and F1 score (F1). All reported results represent the best values ​​from three independent runs, allowing for a comprehensive comparison of classification results with ground truth maps.

[0050] C. Performance Comparison To highlight the performance of the proposed method, this section compares MCAFNet with various state-of-the-art methods from both visual and quantitative perspectives, including: MDL-RS, EndNet, CCR-Net, CALC, ExViT, Sal²RN, HLMamba, MCFNet, and S2F3Net.

[0051] (1) Comparison with different methods: Tables 1 through 4 show the classification results of different methods on the Houston2013, Augsburg, Trento, and MUUFL datasets, respectively. The best results for each class are highlighted in bold. Figures 5 through 6 show the classification results of different methods on the Houston2013, Augsburg, Trento, and MUUFL datasets, respectively. Figure 8 The diagram sequentially displays false-color HSI images, LiDAR images, ground truth maps, and classification results from different methods for each dataset, with colors representing the corresponding categories. Notably, to clearly present the visual results, we have magnified local regions in the classification images, which are marked with red rectangles. Several conclusions can be drawn from the experimental results. Compared to methods relying solely on single-depth feature extraction, CNN-based methods employing multi-feature fusion strategies demonstrate significantly superior performance. For example, methods like Sal... 2The RN approach, which deeply integrates HSI spatial-spectral features and LiDAR elevation features, outperforms fusion methods that rely on single deep feature extraction (such as EndNet and CCR-Net). Cross-modal interaction improves classification performance: the cross-fusion mechanism designed in MDL-RS validates the effectiveness of deep multimodal data interaction, outperforming earlier simple concatenation or summation fusion strategies. Attention mechanisms enhance focus on key features: MCFNet improves the capture of key object features in complex scenes, especially for imbalanced categories, by introducing a multi-head attention mechanism. Multi-scale and decision-level fusion optimizes model adaptability: HLMamba further optimizes classification consistency through its efficient long-range dependency modeling combined with multi-scale feature extraction, demonstrating the importance of combining global context with multi-scale structure for enhancing model robustness. In comparison, the proposed MCAFNet outperforms all comparable methods in OA, AA, Kappa, and F1 scores across four datasets. This advantage stems not only from MCAFNet's combination of CNN's local perception and Mamba's long-range dependency modeling, but also from its unique three-branch architecture, MSFE module, and MBCA module. MCAFNet can simultaneously capture detailed structural and macro-level scene features, achieving deep cross-modal interaction, thus exhibiting stronger adaptability and classification accuracy in complex and ever-changing remote sensing scenarios.

[0052] (2) Quantitative analysis: Tables 1 through 4 present comprehensive quantitative evaluation results on four benchmark datasets. Our proposed MCAFNet method achieves state-of-the-art results on all four datasets. On the Houston2013 dataset, MCAFNet achieves the highest overall accuracy of 96.01%, making it the most accurate among all compared methods. In contrast, methods such as MDL-RS and EndNet exhibit relatively low OA values ​​of 89.22% and 90.07%, respectively. This performance gap may be attributed to their insufficient exploration of cross-modal complementarity. MDL-RS mainly relies on decision-level fusion strategies and fails to achieve deep feature interactions, while EndNet employs a relatively simple fusion method, which may lead to feature confusion between modalities. In contrast, the three-branch architecture of the MCAFNet method effectively overcomes these limitations by establishing dedicated paths for modality-specific feature extraction while providing a structured framework for deep cross-modal interactions, thereby achieving more comprehensive feature representations without loss of information.

[0053] Table 1. Comparison Experiments Using the Houston 2013 Dataset Table 2. Comparison Experiments on the Augsburg Dataset Table 3. Comparison Experiments on the Trento Dataset Table 4. MUUFL Dataset Table 2 shows the robustness of MCAFNet in handling class imbalance, achieving the highest OA (92.39%). Other methods, including CCR-Net and ExViT, exhibited lower performance, with OA values ​​of 88.62% and 89.19%, respectively. One possible explanation for this result is that CCR-Net, which relies heavily on convolutional operations, may not be able to adequately capture long-range dependencies in complex scenes, while ExViT's attention mechanism may not fully utilize the complementary information between HSI and LiDAR modalities. This suggests that the MBCA module in MCAFNet facilitates more effective feature learning through deep cross-modal interaction. Table 3 shows that MCAFNet achieved the highest OA (99.51%) among all methods. Other methods exhibited lower overall accuracy values. The slightly inferior performance of these methods may be due to their insufficient modeling of the complementary relationship between spectral and elevation features. In contrast, MCAFNet's bidirectional cross-attention mechanism effectively utilizes LiDAR elevation information to resolve spectral ambiguity, thus contributing to more accurate classification.

[0054] For the MUUFL dataset in Table 4, MCAFNet demonstrates superior performance with an OA of 90.30%. Comparative methods such as MCFNet and S3F2Net have lower OA values, at 87.56% and 86.80%, respectively. This performance difference may be attributed to MCAFNet's fusion approach, which incorporates multi-scale processing in the MSFE module and performs deep feature interaction in the MBCA module, enabling it to capture more comprehensive and nuanced spectral-spatial features. In contrast, MCFNet's fusion strategy may not fully utilize the multi-scale features of objects, while S3F2Net may face challenges in effectively integrating spatial and spectral information at different scales.

[0055] In summary, our proposed MCAFNet method consistently achieves state-of-the-art performance across most classification categories on all three datasets. Its ability to maintain high accuracy across diverse scenarios, from complex urban environments to agricultural landscapes, demonstrates the effectiveness of its architecture in addressing the challenges of multimodal remote sensing data classification.

[0056] (3) Qualitative analysis: As shown in the classification diagram (Figure 5 to 10) Figure 8 As shown in the figure, MCAFNet demonstrates superior performance on all four datasets, with minimal overall noise and highly accurate classification results. The red rectangles highlight local regions magnified for detailed comparison.

[0057] Traditional CNN-based methods (such as MDL-RS and EndNet) exhibit significant limitations when handling complex urban scenes, manifesting as salt-and-pepper noise and blurred object boundaries. This visual imperfection stems from their limited receptive field and insufficient modeling of long-range dependencies, which is particularly problematic when classifying structurally complex urban features like commercial districts. Similarly, methods relying on simple feature concatenation or early fusion strategies also face challenges in maintaining boundary accuracy within heterogeneous regions—as shown in Figure 6—demonstrated by the classification results of "water bodies" in the Augsburg dataset.

[0058] More advanced methods incorporating attention mechanisms (such as MCFNet and ExViT) show improved performance, but significant noise remains in spectrally similar categories. Figures 7 and 8 show the boundary confusion between "ground" and "road" in the Trento dataset and the classification inconsistency of "mixed surface" in the MUUFL dataset, indicating that unidirectional attention or shallow cross-modal interaction fails to address the feature ambiguity problem. These methods often prioritize one modality while ignoring another, or lack sufficient mechanisms to preserve key spatial details during fusion, frequently introducing excessive smoothing in homogeneous regions and struggling to handle fine-grained spatial details.

[0059] MCAFNet addresses these limitations through its integrated architecture. The multi-scale dilated convolutions of the MSFE module enable comprehensive feature extraction across different object sizes, effectively capturing the fine textures of urban structures and the broad patterns of natural landscapes. Simultaneously, the MBCA module establishes a deep bidirectional interaction between spectral and elevation features, ensuring full utilization of complementary information without sacrificing spatial accuracy. This collaborative design produces classification maps that maintain clear boundaries in complex urban environments while effectively identifying small-scale regions, demonstrating strong adaptability across various remote sensing scenarios. In all cases, MCAFNet achieved consistency and accuracy, validating the effectiveness of its multi-scale cross-attention fusion strategy. Qualitative comparisons show that MCAFNet more accurately delineates the outlines of fine-grained objects such as "buildings" and "sidewalks," further validating its practical value in high-resolution scenes.

[0060] D. Computational complexity analysis Model complexity is a key metric for evaluating model efficiency and resource consumption. We conducted a quantitative analysis on the Houston2013 dataset to compare the number of parameters, floating-point operations, training time, and testing time of various models. As shown in Table 5, MCAFNet contains 403.50K parameters, requires 37.25M FLOPs, takes 360.40 seconds to train, and takes 0.73 seconds to test. Although its parameter count and training time exceed those of some lightweight models (e.g., EndNet, CCR-Net) and are at a mid-level among the ten comparison models, the performance improvement achieved justifies these computational requirements, keeping the parameter count and computational cost within an acceptable range. Regarding testing time, the proposed method demonstrates efficient inference speed. Therefore, the evaluation considering both classification performance and computational complexity shows that our method achieves a reasonable balance between performance gain and resource consumption.

[0061] Table 5. Params, FLOPS, Train time, and Test time in the Houston 2013 dataset E. Ablation Studies To verify the contribution of each proposed module, we conducted a systematic ablation study on four datasets by sequentially removing the MBCA and MSFE modules. The results are shown in Table 6.

[0062] Table 6. Ablation experiments of modules The baseline model, consisting of a three-branch encoder architecture excluding MBCA and MSFE modules, performs the worst across all datasets. For example, its operational accuracy (OA) is only 94.49% and 89.01% on the Houston2013 and MUUFL datasets, respectively. This result highlights the limitations of a simple architecture lacking dedicated mechanisms for deep cross-modal interaction and multi-scale feature extraction.

[0063] Adding the MSFE module alone to the baseline model resulted in a stable performance improvement. A significant gain was observed on the Augsburg dataset, with an AA improvement of 2.61%. This demonstrates that the MSFE module plays a crucial role in enhancing the model's adaptability to scale variations in complex scenes. By employing parallel dilated convolutions and adaptive channel weighting, the MSFE module enables the multimodal branch to capture rich spatial-spectral contextual information across various scales.

[0064] Introducing the MBCA module alone into the baseline model resulted in a more significant performance leap. OA improved by 1.52% on the Houston2013 dataset and by 1.29% on the MUUFL dataset. These substantial improvements confirm the effectiveness of the deep bidirectional interaction mechanism facilitated by the MBCA module. It effectively promotes semantic alignment and information exchange between the HSI branch, LiDAR branch, and multimodal branches, thereby fully utilizing complementary information between modalities.

[0065] Finally, the complete MCAFNet model integrating the MSFE and MBCA modules achieved state-of-the-art performance across all metrics across all datasets. The consistent and comprehensive performance gains compared to variants containing only a single module demonstrate the high complementarity of the MSFE and MBCA modules. The MSFE module provides robust multi-scale feature representations for the fusion branch, laying a rich foundation for interaction. Subsequently, the MBCA module leverages this enhanced representation to perform deep, bidirectional feature fusion across all three branches. This collaborative work of the modules is key to the model's superior classification capabilities.

[0066] F. Parameter Discussion (1) Different Dimensions: Based on the quantitative results shown in Figure 9, we conducted a systematic analysis to investigate the feature dimensions. Impact on final classification performance. Experimental configuration: The values ​​were 32, 64, and 128 to evaluate the model’s sensitivity to this key hyperparameter on four datasets.

[0067] The results show that The optimal choice depends on the specific characteristics of the dataset. For the Houston2013 dataset, a clear trend was observed: larger feature dimensions (specifically...) (=128) yields superior performance across all metrics. This indicates that the complex urban scenes and diverse land cover categories of this dataset benefit from higher-dimensional feature representations, which provide greater capacity for encoding discriminative information.

[0068] A similar preference for larger feature dimensions was observed on the MUUFL dataset. =128 again yielded the best result. This means that a sufficiently large feature space is needed to fully model the spectral-spatial details of the campus environment. In contrast, the model's performance on the Augsburg dataset varies. The values ​​remain relatively stable, showing only minor fluctuations. Notably, the classification accuracy exhibits significant robustness on the Trento dataset, with negligible variations across the test dimensions. This insensitivity suggests that even with a more compact representation, the model effectively captures the essential features used for Trento agricultural landscape classification.

[0069] Taking overall performance into consideration, The feature dimension of 128 achieves state-of-the-art or highly competitive results on most datasets without introducing significant computational overhead.

[0070] (2) Different image patch sizes: We further investigated the impact of input image patch size on the model's classification performance. A comprehensive analysis was conducted by varying the image patch size from 3×3 to 15×15 with a step size of 2. The overall accuracy trends for the four datasets are summarized in Figure 10.

[0071] Experimental results show that the optimal image patch size depends on the specific dataset, which can be attributed to different spatial features, scene complexity, and land cover distribution. For the MUUFL dataset, the highest classification accuracy was achieved at an image patch size of 7×7, as this compact size effectively captures the ubiquitous local spatial details in the scene. The model exhibits significant robustness on the Trento dataset, with minimal accuracy fluctuations across image patch sizes ranging from 7×7 to 13×13. This indicates that the model is less sensitive to spatial context scales in this specific agricultural environment. In contrast, the Houston2013 and Augsburg datasets achieved peak performance at an image patch size of 11×11. For these scenes, this moderate size provides an optimal balance, offering sufficient spatial context to distinguish complex land cover categories without introducing excessive irrelevant information that could serve as noise. While the ideal image patch size exhibits some dataset-specific variations, the 11×11 size consistently provides strong and stable performance across all datasets. It either matches or comes very close to the optimal result in each case while maintaining computational efficiency.

[0072] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a land cover classification method combining hyperspectral and lidar data, and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0073] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of a land cover classification method combining hyperspectral and lidar data, and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0074] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0075] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0076] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0077] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for land cover classification combining hyperspectral and lidar data, characterized in that, Includes the following steps: Acquire hyperspectral image data and lidar elevation data of the ground features to be processed; A three-branch parallel structure is used to extract features from the hyperspectral image data and the lidar elevation data. The three-branch parallel structure includes a hyperspectral image branch, a lidar branch, and a multimodal branch. In the hyperspectral image branch, the hyperspectral image data is subjected to three-dimensional convolution and bidirectional scanning along the spectral dimension to extract global spectral spatial features; In the lidar branch, the lidar elevation data undergoes multi-level convolution processing to extract elevation structure features; In the multimodal branch, the hyperspectral image data and the lidar elevation data are fused in the early stage and processed by the multi-scale feature enhancement module to extract multimodal fusion features. The multi-scale feature enhancement module achieves multi-scale contextual information capture through parallel dilated convolutions with different dilation rates. A multi-head bidirectional cross-attention module is used to perform deep fusion processing on the global spectral spatial features, the elevation structure features, and the multi-scale fusion features. The deep fusion processing includes performing bidirectional cross-attention calculation on any two features and aggregating all interaction results to generate the final fused feature vector. The final fused feature vector is input into the classifier, which outputs the land cover classification results corresponding to the hyperspectral image data and the lidar elevation data.

2. The method according to claim 1, characterized in that, The step of extracting global spectral spatial features specifically includes: The input hyperspectral image data is converted into a three-dimensional feature cube. Feature extraction is performed using three-dimensional depthwise separable convolutional blocks. The calculation process of the three-dimensional depthwise separable convolutional blocks is as follows: ； ； in, For depthwise convolution output, For pointwise convolution output, DWCov3×3×3 is a 3D depthwise convolution, Conv1×1×1 is a 1×1×1 pointwise convolution, BN3D is a 3D batch normalization, and ζ(·) is the GELU activation function. The pointwise convolution output Pooling and activation processes are performed to obtain post-processed features. ; The post-processing features are obtained using a 3D scanning module. The 3D scanning module performs processing on the post-processed features. After serialization along the spectral dimension, a bidirectional scan is performed, both forward and backward, and the results of the bidirectional scan are averaged to obtain the spectral scan features. ; The spectral scanning features are layer normalized and compared with the post-processed features. Perform residual connections to obtain the global spectral spatial features. The calculation process is as follows: 。 3. The method according to claim 1, characterized in that, The steps for extracting elevation structure features specifically include: The lidar elevation data is used as the initial input. Layered feature extraction is performed through N consecutive convolutional blocks, where the output feature of the i-th convolutional block is... The calculation process is as follows: ； Where Conv3×3 is a 3×3 two-dimensional convolution, BN is two-dimensional batch normalization, and ξ(·) is the ReLU activation function. The output features of the (i-1)th convolutional block; The output of the last convolutional block is used as the elevation structure feature. The elevation structural features The dimension and the global spectral spatial features The dimensions remain consistent.

4. The method according to claim 1, characterized in that, In the step of extracting multi-scale fusion features, the specific process of early fusion includes: The hyperspectral image data With the lidar elevation data Stitching along the channel dimension; The concatenated data is then projected and nonlinearly transformed through a 1×1 two-dimensional convolutional layer. The transformed features are compared with the original hyperspectral image data. Perform residual connections to obtain early fusion features. The calculation process is as follows: ； Where [·,·] represents the concatenation operation along the channel dimension, Conv1×1 is a 1×1 two-dimensional convolution, and ξ(·) is the ReLU activation function.

5. The method according to claim 4, characterized in that, After obtaining the aforementioned early fusion characteristics Subsequently, the step of extracting multimodal fusion features through multi-scale feature enhancement module processing specifically includes: Regarding the early fusion features Intermediate features are obtained by sequentially processing convolutional blocks and two-dimensional depthwise separable convolutional blocks. ; The intermediate features The input is fed into the multi-scale feature enhancement module, which captures multi-scale contextual information through N parallel dilated convolutional branches with different dilation rates k. The output of the k-th scale branch... The calculation process is as follows: ； Where Conv3×3,dilation=k represents a 3×3 convolution with a dilation rate of k; The outputs of all scale branches are concatenated to obtain the multi-scale fused features. The calculation process is as follows: ； For the multi-scale fusion features The multimodal fusion features are obtained by performing convolutional enhancement and channel attention weighting. The calculation process for the channel attention weighting is as follows: ； ； in, The features are enhanced by convolution. GAP is global average pooling, σ is the Sigmoid activation function, and ζ is the GELU activation function. This indicates element-wise multiplication along the channel dimension.

6. The method according to claim 1, characterized in that, The deep fusion process includes a step of performing bidirectional cross-attention calculation on any two features, including the global spectral spatial features. and the aforementioned elevation structural features The steps for performing bidirectional cross-attention calculation specifically include: The input global spectral spatial features and the aforementioned elevation structural features Perform layer normalization; Each normalized feature is passed through a linear projection layer to generate a query vector Q, a key vector K, and a value vector V. Perform bidirectional cross-attention computation, wherein the first-way cross-attention computation uses query vector and key vector Sum value vector To enable interaction, the second-way cross-attention calculation is used. query vector and key vector Sum value vector Interact; The output of the first cross-attention calculation is compared with its corresponding query vector. Perform a residual connection, and combine the output of the second-direction cross-attention calculation with its corresponding query vector. Perform residual connections to preserve the original information of each component, resulting in a bidirectional interactive output representation. .

7. The method according to claim 6, characterized in that, The deep fusion process also includes the following steps: Sequentially, global spectral spatial features Multi-scale fusion features Global spectral spatial characteristics With elevation structural features Multi-scale fusion features With elevation structural features These three feature pairs undergo the bidirectional cross-attention calculation. The results of the bidirectional cross-attention calculations for all three feature pairs are summed to obtain the aggregated features. The aggregated features are then subjected to global average pooling to convert them into a one-dimensional vector, resulting in the final fused feature vector. .

8. The method according to claim 1, characterized in that, The step of inputting the final fused feature vector into the classifier specifically includes: The final fused feature vector The input is fed into a classifier consisting of a dropout layer and a linear layer; The classifier outputs the predicted class probability for each pixel through a linear transformation; During the model training phase, the cross-entropy loss function is used to calculate the loss between the predicted probability and the true label, and the model parameters are updated through backpropagation until the model converges.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of a land cover classification method combining hyperspectral and lidar data as described in any one of claims 1-8.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of a land cover classification method combining hyperspectral and lidar data as described in any one of claims 1-8.

Images