Ground feature classification method based on combination of hyperspectral image and laser radar

Abstract

The invention belongs to the technical field of remote sensing, and provides a surface feature classification method based on combination of a hyperspectral image and a laser radar. The method comprises the following steps: acquiring a hyperspectral image and laser radar data; labeling ground feature categories in the processed data, and making a label file; constructing a ground object target detection model and performing training according to the data and the label file; the model comprises a multi-scale spectrum-space sparse coding module used for extracting hyperspectral sparse features, a multi-scale geometric structure sparse coding module used for extracting laser radar geometric sparse features, and a competitive sparse selection module used for multi-scale feature dynamic fusion. And finally, utilizing the trained model to generate a ground feature classification result. According to the terrain classification method, the problem of neglecting modal sparsity difference and feature redundancy is solved through a'modal specificity sparse modeling + multi-scale competitive selection 'mechanism, and the accuracy and robustness of multi-source remote sensing data terrain classification in a complex scene are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of remote sensing technology, specifically to a method for classifying ground features based on the combination of hyperspectral images and lidar. Background Technology

[0002] In recent years, the rapid advancement of multi-source remote sensing technology has greatly enriched the data sources for land surface monitoring. The comprehensive utilization of remote sensing data from different modalities has become a core research direction in land cover classification and land cover identification. Among these, the fusion of hyperspectral imaging (HSI) and lidar (LiDAR) data has attracted significant attention due to its remarkable complementarity. HSI provides continuous spectral response information, enabling precise characterization of the reflectivity of land cover features through wavelength variations, thus effectively distinguishing different materials and biochemical components. LiDAR technology can capture the three-dimensional geometric structure and elevation distribution of the land surface with high precision, compensating for the limitations of hyperspectral data in spatial structure description. The fusion of these two technologies can improve the accuracy of identifying land cover features in complex scenes across spectral, spatial, and structural dimensions. With the development of deep learning technology, fusion methods based on convolutional neural networks (CNN), Transformers, and state-space models (such as Mamba) have achieved breakthrough progress. CNN excels at extracting local features, Transformers achieve global context modeling through self-attention mechanisms, while Mamba maintains linear computational complexity while possessing the ability to model long sequence dependencies.

[0003] However, existing joint classification methods using hyperspectral and lidar still have significant limitations: Isomorphic feature extraction ignores modal differences: Existing methods often use isomorphic coding structures or shared parameter spaces to map different modes to a unified feature domain. This "isomorphic processing" ignores the fundamental physical difference between the significant "spectral sparsity" of hyperspectral data and the "geometric / gradient sparsity" of lidar data. This approach often leads to the loss of spectral details or excessive smoothing of geometric information, resulting in redundant and ambiguous feature representations. Multi-scale fusion lacks a competitive selection mechanism: Ground targets vary greatly in spatial scale, but existing multi-scale fusion methods (such as simple cascading or weighted summation) lack an explicit competitive selection mechanism. This easily leads to the indiscriminate accumulation of features at different scales, making it difficult for the model to adaptively focus on the most discriminative key scale in complex scenes, resulting in information redundancy and interference.

[0004] Therefore, it is evident that how to fully utilize modality-specific sparse priors during the fusion process and establish an effective multi-scale competitive screening mechanism are key issues for further improving the joint classification accuracy of hyperspectral and lidar systems. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a land cover classification method based on the combination of hyperspectral images and lidar, in order to solve or partially solve the problem in the prior art that ignores modal sparsity differences and feature redundancy, which affects the accuracy of land cover classification.

[0006] In a first aspect, embodiments of the present invention provide a method for land cover classification based on the combination of hyperspectral images and lidar, comprising the following steps: S1: Acquire hyperspectral image data and lidar data and perform preprocessing; S2: Label the land cover categories in the processed hyperspectral image and lidar data, and create a label file; S3: Construct a ground target detection model and train it based on the hyperspectral image, lidar data, and the label file. The ground target detection model includes a multi-scale spectral-spatial sparse coding module (MS-SSSE) for extracting hyperspectral sparse features, a multi-scale geometric sparse coding module (MS-GSSE) for extracting lidar geometric sparse features, and a competitive sparse selection module (CSS) for dynamic fusion of multi-scale features. The competitive sparse selection module includes: a visual selective state space block (VSS Block), used to process small, medium, and large-scale features separately to capture long-distance spatial dependencies within the features; and a spatial competitive selection sub-module (SCS Block), used to establish explicit competitive relationships between features at different scales. S4: The trained ground object detection model is used to process the ground object data to be classified. After capturing global dependencies through the visual selective state space block (VSS Block) and performing scale competition through the spatial competition selection submodule (SCSBlock), the location and category of the ground object are generated.

[0007] The beneficial effects of the above embodiments are as follows: the local object classification method integrates the spectral information of hyperspectral images with the spatial structure information of lidar, and combines the multi-scale spectral-spatial sparse coding module (MS-SSSE), the multi-scale geometric structure sparse coding module (MS-GSSE), and the competitive sparse selection module (CSS). It uses the trained model to process the data to be classified, and after capturing global dependencies through the VSS Block and performing scale competition through the SCS Block, it outputs the location and category of the ground object target. This solves the problems of insufficient modality-specific expression and multi-scale feature redundancy in the existing technology, and improves the accuracy and robustness of ground object classification in complex scenes.

[0008] According to a specific implementation of an embodiment of the present invention, the preprocessing in step S1 includes: Hyperspectral data preprocessing: The hyperspectral image data is subjected to radiometric calibration, atmospheric correction, geometric correction and format conversion in sequence; LiDAR data preprocessing: The LiDAR data is sequentially subjected to point cloud filtering, point cloud classification, elevation normalization, rasterization, and format conversion; Overall data preprocessing: The preprocessed lidar data and hyperspectral images are registered with coordinate system one and sub-pixel level.

[0009] Beneficial effects: Radiometric calibration eliminates sensor noise, atmospheric correction removes atmospheric scattering and absorption interference, and geometric correction ensures spatial positioning accuracy. Combined with point cloud filtering and registration processing, data quality and modal consistency are improved, providing reliable input for subsequent model training.

[0010] According to a specific implementation of an embodiment of the present invention, the annotation in step S2 includes: manually or semi-automatically annotating the land cover categories in the hyperspectral image and lidar data using the ROI tool based on ENVI software to generate a label file with the same spatial resolution as the original data; the label file is stored in .mat format, and the hyperspectral image data and lidar data share the same label file.

[0011] Beneficial effects: High-precision pixel-level annotation is achieved through ROI tools, ensuring the spatial correspondence between labels and multimodal data, reducing data redundancy by sharing label files, and improving model training efficiency and annotation consistency.

[0012] According to a specific implementation of an embodiment of the present invention, the multi-scale spectral-spatial sparse coding module (MS-SSSE) and the multi-scale geometric sparse coding module (MS-GSSE) in step S3 are specifically constructed as follows: MS-SSSE module: For hyperspectral data, it first uses one-dimensional grouped convolution to decompose the spectral subspace and uses the spectral attention mechanism to adaptively select discriminative channels; then it uses multi-scale depth separable convolution to extract spatial patterns and embeds a learnable soft thresholding layer to achieve feature sparsity.

[0013] The MS-GSSE module, designed for LiDAR data, maps the data to a feature space via a projection layer. It employs depth-separable directional strip convolutions to capture geometric contours in both the horizontal and vertical directions, and combines this with learnable soft-thresholding layers to enhance the sparse representation of the geometric structure. Beneficial effects: MS-GSSE achieves modality-specific "native feature sparsity" through soft-thresholding layers and grouped convolutions, effectively removing spectral redundancy and noise from hyperspectral data. MS-GSSE utilizes directional convolutions to accurately capture the edge and contour information of LiDAR data; the combination of these two approaches enables physically interpretable sparse feature modeling at the architectural level.

[0014] According to a specific implementation of an embodiment of the present invention, the Visual Selective State Space Block (VSS Block) included in the Competitive Sparse Selection Module (CSS) in step S3 specifically includes: The sequence unrolling unit, based on the two-dimensional selective scanning mechanism (SS2D), unrolls the input image block into a feature sequence along four directions: top-left to bottom-right, bottom-right to top-left, top-right to bottom-left, and bottom-left to top-right. The state-space modeling unit processes the unrolled feature sequence using the S6 block, capturing long-range dependencies through state equations and gating mechanisms. The sequence merging unit re-merges the processed four-directional sequences into image features and performs feature transformation using LayerNorm and Feedforward Neural Network (FFN). Beneficial effects: The VSS Block, through four-directional scanning and a state-space model, achieves global receptive field construction while maintaining linear computational complexity, effectively overcoming the limitations of traditional CNNs' limited receptive field and the high computational overhead of Transformers, significantly improving the model's efficiency in capturing long-range spatial dependencies.

[0015] According to a specific implementation of an embodiment of the present invention, the Spatial Competition Selection Submodule (SCS Block) included in the Competitive Sparse Selection Module (CSS) in step S3 specifically includes: The competitive weight generation unit receives features from two different scales and the fused features, and generates pixel-level spatial competitive weights in the channel dimension through 1×1 convolution and the Softmax function. The feature weighting unit uses the generated competitive weights to weight the input features at the two scales respectively, to enhance the response of discriminative regions and suppress redundant regions. The global dynamic balancing unit generates learnable global fusion weights through global average pooling (GAP) and multilayer perceptron (MLP) to balance the contribution ratios between different competitive groups. Beneficial effects: SCS Block establishes an explicit competition mechanism between multi-scale features, avoiding feature redundancy caused by simple averaging or superposition in traditional multi-scale fusion. This allows the model to adaptively focus on the most discriminative scale based on ground features, significantly improving classification accuracy in complex scenes.

[0016] Secondly, embodiments of the present invention provide a ground cover classification device based on the combination of hyperspectral images and lidar, comprising: The acquisition module is used to acquire hyperspectral image data and lidar data and perform preprocessing; the annotation module is used to annotate the land cover categories in the processed hyperspectral images and lidar data and create label files. A construction module is used to build a ground target detection model and train it based on the hyperspectral image, lidar data, and the label file. The ground target detection model includes a multi-scale spectral-spatial sparse coding module (MS-SSSE) for extracting hyperspectral sparse features, a multi-scale geometric structure sparse coding module (MS-GSSE) for extracting lidar geometric sparse features, and a competitive sparse selection module (CSS) for dynamic fusion of multi-scale features. The competitive sparse selection module includes: a visual selective state space block (VSS Block), used to process small, medium, and large-scale features separately to capture long-distance spatial dependencies within the features; and a spatial competitive selection sub-module (SCS Block), used to establish explicit competitive relationships between features at different scales. The generation module is used to process the ground object data to be classified using the trained ground object detection model. After capturing global dependencies through the visual selective state space block (VSS Block) and performing scale competition through the spatial competition selection submodule (SCS Block), the location and category of the ground object are generated.

[0017] The present invention provides a method and apparatus for land cover classification based on the combination of hyperspectral images and lidar, which has at least the following technical effects: First, the local object classification method constructs a fusion mechanism of "modality-specific sparse modeling + multi-scale competitive selection". It combines the MS-SSSE module and the MS-GSSE module to extract spectral sparse features of hyperspectral data and geometric sparse features of lidar data, respectively. It also achieves architecture-level sparsity constraints through an embedded soft threshold layer. This effectively solves the problems of modality-specific loss and feature redundancy caused by isomorphic feature extraction strategies in existing technologies, and significantly improves the physical interpretability and discriminative power of the features.

[0018] Second, the local object classification method introduces a competitive sparse selection (CSS) module, which establishes explicit competitive relationships among multi-scale features through hierarchical cascaded spatial competitive selection (SCS) units. This mechanism can dynamically suppress redundant scales and enhance the response weights of key scales according to the characteristics of ground objects, avoiding the accumulation of invalid information caused by traditional multi-scale fusion methods (such as simple feature overlay or averaging), thereby achieving more accurate scale-adaptive fusion in complex scenarios.

[0019] Third, the local object classification method integrates the Visual Selective State Space Block (VSS Block) in the fusion stage, and uses the two-dimensional selective scanning (SS2D) mechanism to capture long-distance spatial dependencies. While maintaining linear computational complexity (reducing the computational overhead of Transformer), it breaks through the limitation of the local receptive field of CNN. In addition, the model shows excellent robustness under small sample conditions and can generate classification results with clear boundaries and strong spatial continuity. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0021] Figure 1 A flowchart of a land cover classification method based on the combination of hyperspectral images and lidar provided in an embodiment of the present invention is shown; Figure 2 The diagram shows the structural block diagram of the ground target detection model provided in the embodiment of the present invention; Figure 3 The diagram shows a structural block diagram of a land cover classification device based on the combination of hyperspectral images and lidar provided in an embodiment of the present invention. Detailed Implementation

[0022] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0023] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0024] Figure 1 A flowchart illustrating the steps of a land cover classification method based on the combination of hyperspectral images and lidar provided in this embodiment of the invention is shown below. Figure 1 The method includes the following steps: S1: Acquire raw hyperspectral images and lidar data and perform preprocessing.

[0025] Hyperspectral data preprocessing includes: radiometric calibration, atmospheric correction, and geometric correction; LiDAR data preprocessing includes: point cloud registration, coordinate transformation, and intensity correction; The overall data preprocessing includes: accurately registering the lidar point cloud with the hyperspectral image.

[0026] Specifically: 1.1 Hyperspectral data: 1.1.1: Data Acquisition: Raw hyperspectral remote sensing image data is acquired from the airborne platform. In this embodiment, the ITRES CASI-1500 airborne hyperspectral imager is used. This sensor has extremely high spectral resolution (144 consecutive bands) and moderate spatial resolution (2.5 meters), covering the visible to near-infrared band (0.38-1.05 μm), making it very suitable for fine classification tasks of complex urban land features.

[0027] 1.1.2: Data Processing: After data download, ENVI software is used to systematically preprocess the airborne hyperspectral images, including key steps such as radiometric calibration, atmospheric correction, and geometric correction. First, radiometric calibration is performed, converting the digital quantization (DN) values ​​recorded by the sensor into physically meaningful radiance values. This requires applying the sensor's calibration coefficient file. Next, atmospheric correction is performed using ENVI's FLAASH module or ATCOR-4 module (specifically for airborne data). By inputting parameters such as flight altitude, atmospheric mode, and aerosol type, the effects of atmospheric scattering and absorption are eliminated to obtain surface reflectance data. For the shadowing effect in urban areas, additional shadow detection and compensation processing is required. Finally, geometric fine correction is performed, using ground control point and flight attitude data to ensure the spatial positioning accuracy of the images.

[0028] 1.1.3: Data Format Conversion and Output: After processing, the hyperspectral data is exported to a standard format. MATLAB is used to read the ENVI standard format files (.img and .hdr file pairs) and convert them to .mat format for subsequent deep learning model processing. During the conversion process, complete metadata information needs to be preserved, including the center wavelengths of the 144 bands, full width at half maximum (FWHM), spatial resolution (2.5 meters), and projected coordinate system.

[0029] 1.2 LiDAR Data: 1.2.1: Data Acquisition: Raw point cloud data is acquired from the airborne lidar platform. In this embodiment, data is obtained from the Optech Gemini airborne lidar ranging system. This system operates in the 1064nm near-infrared band and has high-density point cloud acquisition capabilities and accurate elevation measurement. It can generate high-quality digital surface models (DSMs), which are very suitable for urban 3D structure extraction and ground feature height information acquisition.

[0030] 1.2.2: Data Processing: After data acquisition, preprocessing is performed using professional point cloud processing software. First, point cloud filtering is performed to remove noise points and outliers, employing statistical outlier removal algorithms or filtering methods based on local point density. Next, point cloud classification is performed, categorizing the raw point cloud into ground points, vegetation points, building points, etc. This step is crucial for generating accurate digital elevation models (DEMs) and digital surface models (DSMs). Then, elevation normalization is performed, converting ellipsoidal heights to orthographic heights or geoid heights; in this example, the Geoid 2012A model is referenced. Finally, the irregularly distributed point cloud data is rasterized using interpolation algorithms to generate a DSM product with the same spatial resolution (2.5 meters) as the hyperspectral image.

[0031] 1.2.3: Data Format Conversion and Output: Similarly, after processing, the LiDAR data is converted to .mat format for easy processing by subsequent deep learning models.

[0032] 1.3: Overall Data Processing: Accurately register the lidar point cloud with the hyperspectral image; First, coordinate system 1 is established to ensure that both types of data use the same projected coordinate system. Then, precise registration is performed, achieving sub-pixel level registration accuracy by extracting common feature points or using a mutual information registration algorithm. Finally, MATLAB is used to convert the processed DSM data to .mat format and save it as a single-band elevation matrix.

[0033] S2: Label the land cover categories in the processed hyperspectral images and lidar data, and create a label file; Based on the actual conditions of the target area, land features can be classified into various categories, such as water bodies, farmland, buildings, roads, and forests. ENVI software is used to annotate the hyperspectral image. ENVI provides various tools, such as the ROI (Region of Interest) tool, which allows for manual or semi-automatic annotation of land feature categories in the target area. Each category of region is assigned a specific label, forming a label file with the same dimensions as the original hyperspectral image. When creating the labels, it is essential to ensure that the labels match the spatial resolution of the image, and that the label for each pixel reflects the land feature category to which that pixel belongs.

[0034] After annotation, the label file is exported and converted to .mat format using MATLAB. This conversion step ensures that the label file can be read and processed along with the hyperspectral and LiDAR data, facilitating subsequent training. The .mat format data not only stores label information but also includes corresponding category labels and assignment rules, ensuring that the correspondence between the label file and the hyperspectral image and LiDAR data remains intact. It is worth noting that the hyperspectral and LiDAR data share the same label file.

[0035] S3: Construct a ground target detection model and train it based on hyperspectral image data, lidar data, and label files. The ground target detection model includes a multi-scale spectral-spatial sparse coding module (MS-SSSE), a multi-scale geometric sparse coding module (MS-GSSE), and a competitive sparse selection module (CSS). In this embodiment, the overall architecture of the ground object detection model (CSS-Mamba) is constructed based on the principle of "modality-specific sparse modeling + multi-scale competitive selection". The process includes multi-scale feature extraction (MS-SSSE and MS-GSSE), hierarchical competitive sparse selection fusion (CSS), and final classification. This architecture aims to solve the redundancy problem caused by isomorphic feature extraction and the lack of explicit selection mechanism in multi-scale fusion.

[0036] Phase 1: Modality-specific sparse feature extraction; The model first processes hyperspectral and lidar image patches of three different scales (small, medium, and large) in parallel through a dual-branch network.

[0037] For hyperspectral data, the model employs a multi-scale spectral-spatial sparse coding module (MS-SSSE). This module is designed based on the physical prior that hyperspectral data possesses rich spectral dimensions and exhibits spectral sparsity. Specifically, it first performs spectral subspace decomposition using one-dimensional group convolutions (Group Conv) to capture independent spectral response patterns in each group; then, a lightweight spectral attention mechanism is introduced to adaptively select the spectral channels most discriminative for the classification task; next, it enhances local spatial patterns using depthwise separable convolutions at different scales, and explicitly performs sparse coding at the architecture level through an embedded learnable soft thresholding layer to filter out redundant noise and generate spectral-spatial features with "native sparsity".

[0038] For lidar data, since its discriminative information is concentrated on spatial structure and elevation abrupt changes, the model employs a multi-scale geometric sparse coding module (MS-GSSE). This module designs depth-separable directional strip convolutions, utilizing asymmetric convolution kernels to extract the edge contours and morphological features of ground features in both the horizontal and vertical directions. The extracted features are also processed by an embedded soft thresholding layer to preserve geometrically responsive edges and suppress background noise in smooth regions, thereby achieving a sparse representation of the geometric domain.

[0039] Phase Two: Competitive Sparse Selection and Dynamic Fusion; The multi-scale modal features processed by the above modules enter the competitive sparse selection module (CSS). This module integrates the visual selective state space block (VSS Block) and the spatial competitive selection sub-module (SCS Block) to establish explicit scale competition relationships and capture global dependencies.

[0040] First, the VSS Block is used to enhance the modeling of internal long-range dependencies of features at various scales. It introduces a two-dimensional selective scan mechanism (SS2D), which unfolds image features into sequences along four directions: top left to bottom right, bottom right to top left, etc., and processes them using a state-space model (Mamba). This design not only inherits the global receptive field advantage of the Transformer, but also reduces the computational complexity from quadratic to linear, greatly improving the efficiency of processing large-scale remote sensing data.

[0041] Subsequently, the SCS Block performs competitive fusion across multiple scales using a hierarchical strategy. Instead of simply stacking multi-scale features, the model employs a "neighboring scale competition" mechanism. The first level of competition involves spatial competition between large-scale and medium-scale enhanced features. The SCS module generates pixel-level competitive weights in the channel dimension through convolution and the Softmax function, dynamically suppressing redundant scale responses and enhancing the feature performance at discriminative scales.

[0042] The second level of competition involves a similar spatial competition between mesoscale and small-scale augmentation features.

[0043] Finally, adaptive global balancing is performed. The feature outputs after two levels of competition are weighted and summed using a learnable global fusion weight. This weight is determined by the sigmoid activation values ​​generated by the global average pooling layer and the multilayer perceptron (MLP), aiming to dynamically balance the contributions of different competing groups (such as the large-to-medium scale group and the medium-to-small scale group) according to the characteristics of the input scene, thereby flexibly adapting to complex and ever-changing classification scenarios.

[0044] Finally, the fused feature vectors are input into the classifier, which outputs the predicted land cover category. The entire process removes redundancy through sparse constraints and optimizes the scale through a competitive mechanism, significantly improving the model's accuracy in land cover recognition in complex scenes.

[0045] Specifically: 3.1: Using the scipy library, the hyperspectral data, lidar data, and tag files generated in steps one and two are read and loaded into Python. The scipy.io.loadmat function is used to read the .mat file and convert the data into NumPy arrays for processing.

[0046] 3.2: Use the PyTorch framework to write code and build a ground target detection model based on joint classification of two-stage hyperspectral and lidar.

[0047] 3.3: The ground target detection model was trained using hyperspectral data, LiDAR data, and tag files. Supervised learning was employed during training, optimizing the model by minimizing the cross-entropy loss function. The Adam optimizer was used for training, and the model's performance on the validation set was periodically evaluated during training. Hyperparameters (such as learning rate, batch size, and number of network layers) were adjusted to prevent overfitting.

[0048] 3.4: Validate the trained ground object detection model.

[0049] After training, the trained multimodal fusion model was used to perform land cover classification validation on hyperspectral and LiDAR data in the test set. To intuitively evaluate the model performance, the classification results were visualized using Python's Matplotlib library. Specifically, the model's output category predictions were spatially registered and overlaid with the original hyperspectral images to generate a classification map containing land cover category labels and spatial location information, allowing for a direct observation of the model's recognition performance and classification accuracy for different land cover targets.

[0050] Specifically: 3.4.1: Standardize the hyperspectral imagery and LiDAR-DSM data in the test set according to the preprocessing workflow described in step S1, including normalization, spatial registration, and multi-scale patch extraction. Use the scipy.io.loadmat function to load the .mat format test data file, ensuring that the data format is consistent with the model input requirements.

[0051] 3.4.2: A sliding window strategy is used to perform pixel-by-pixel classification of the entire test image. For each pixel to be classified, its multi-scale neighborhood patch (containing spectral-spatial features) centered on that pixel and the corresponding LiDAR-DSM elevation information are extracted. The extracted multi-modal features are then input into the trained CSS-Mamba model for class prediction. Finally, the prediction results of all pixels are combined into a complete classified image.

[0052] 3.4.3: The model outputs the probability distribution of 15 categories through forward propagation. The categories include: Healthy Grass, Stressed Grass, Synthetic Grass, Trees, Soil, Water, Residential, Commercial, Road, Highway, Railway, Parking Lot1, Parking Lot2, Tennis Court, and Running Track.

[0053] 3.4.4: A confidence threshold strategy is used for class determination: when the predicted probability of a certain class exceeds a preset threshold of 0.6, the pixel is directly labeled as belonging to that class; for pixels whose probabilities for all classes do not reach the threshold, the class with the highest probability is selected as the final prediction result. To further improve the spatial continuity and smoothness of the classification map, a 5×5 window median filter is used to perform spatial post-processing on the preliminary classification results, effectively suppressing salt-and-pepper noise and maintaining the clarity of ground feature boundaries.

[0054] 3.4.5: Use the Matplotlib library to visualize the classification results. Fifteen land cover categories were assigned distinct RGB color values, following the visual conventions of land cover types: Healthy Grassland (bright green, RGB: 0,255,0), Crushed Grassland (yellowish-green, RGB: 173,255,47), Artificial Grassland (cyan-green, RGB: 0,255,255), Trees (dark green, RGB: 0,128,0), Soil (brown, RGB: 139,69,19), Water Bodies (blue, RGB: 0,0,255), Residential Area (pink, RGB: 255,192,203), Commercial Area (purple, RGB: 128,0,128), Roads (gray, RGB: 128,128,128), Highways (dark gray, RGB: 64,64,64), Railways (black, RGB: 0,0,0), Parking Lot 1 (light gray, RGB: 192,192,192), Parking Lot 2 (orange, RGB: (255,165,0), tennis court (light green, RGB: 144,238,144), running track (red, RGB: 255,0,0). The generated classification map is displayed side by side with the original hyperspectral pseudocolor map (formed by band combination) and LiDAR-DSM grayscale map to form a comparative analysis group, so as to intuitively evaluate the classification performance and spatial consistency of the model on different land cover types.

[0055] 3.4.6: Accuracy evaluation was performed on 14,586 samples in the test set. The confusion matrix, overall accuracy (OA), average accuracy (AA), and Kappa coefficient were calculated to evaluate the model's recognition performance for 15 types of land features.

[0056] To further evaluate the model's performance, t-distributed random neighborhood embeddings (t-SNEs) can be visualized in two-dimensional space to analyze the feature distribution and the model's classification performance across different land cover categories. By comparing the real labels with the model's predictions, the model's accuracy can be quantitatively assessed, and potential false positives or false negatives can be analyzed. The model can then be revised based on the validation results.

[0057] S4: The trained ground object detection model is used to process the ground object data to be classified. After capturing global dependencies through the Visual Selective State Space Block (VSS Block) and performing scale competition through the Spatial Competition Selection Submodule (SCS Block), a visualization image of the ground object location and category is generated.

[0058] The target detection output is the category and probability distribution of each ground object. If the probability exceeds the set threshold, the ground object is identified as the corresponding category. This process is similar to steps 3.4.4 and 3.4.5, thereby generating a visual image of the location and category of the ground object.

[0059] The local object classification method has been validated using the Houston dataset. Practical verification shows that the proposed method achieves 95.29% OA, 96.03% AA, and 94.90% Kappa on the Houston dataset, particularly achieving completely correct classification for the Stressed Grass, Soil, and Running Track categories. This demonstrates the significant technical advantages of the local object classification method in joint hyperspectral and LiDAR image target detection, effectively improving target classification accuracy and optimizing computational efficiency, thus showing promising application prospects.

[0060] The embodiments of the present invention have the following technical effects: First, the local land cover classification method achieves modality-specific sparse modeling through a multi-scale spectral-spatial sparse coding module (MS-SSSE) and a multi-scale geometric sparse coding module (MS-GSSE), combined with an embedded soft thresholding layer. This effectively solves the problems of neglecting modal physical characteristics and feature redundancy caused by isomorphic processing in traditional fusion methods. Simultaneously, through the spatial competition selection mechanism (SCS) in the competitive sparse selection module (CSS), explicit competition and dynamic selection among multi-scale features are achieved, solving the "averaging" problem caused by the smoothing of weight distribution in traditional multi-scale fusion. This significantly improves the discrimination ability of key-scale features and the accuracy of land cover classification in complex scenes.

[0061] Secondly, compared with traditional methods, this local object classification method introduces a two-dimensional selective scanning mechanism using Visual Selective State Space Blocks (VSS Blocks), successfully reducing the computational complexity of global context modeling from a quadratic Transformer complexity to a linear one. This significantly reduces computational resource consumption while retaining strong long-range dependency capture capabilities. Furthermore, experimental results demonstrate that this method maintains significantly higher classification accuracy than existing methods even with small sample conditions (e.g., only 20 training samples per class), exhibiting strong generalization ability and robustness, effectively addressing the challenge of scarce labeled samples in remote sensing tasks.

[0062] Third, the local land cover classification method can be integrated into remote sensing data processing systems to provide high-precision land cover classification services for agricultural monitoring, geological disaster early warning, etc., and has excellent industrial applicability.

[0063] Figure 3This is a structural block diagram of a land cover classification device based on the combination of hyperspectral images and lidar, according to an embodiment of the present invention. The device includes: The acquisition module is used to acquire hyperspectral image data and lidar data and perform preprocessing. The annotation module is used to annotate the land cover categories in the processed hyperspectral images and lidar data, and create label files; A construction module is used to build a ground target detection model and train it based on the hyperspectral image, lidar data, and the label file. The ground target detection model includes a multi-scale spectral-spatial sparse coding module (MS-SSSE) for extracting hyperspectral sparse features, a multi-scale geometric structure sparse coding module (MS-GSSE) for extracting lidar geometric sparse features, and a competitive sparse selection module (CSS) for dynamic fusion of multi-scale features. The competitive sparse selection module includes: a visual selective state space block (VSS Block), used to process small, medium, and large-scale features separately to capture long-distance spatial dependencies within the features; and a spatial competitive selection sub-module (SCS Block), used to establish explicit competitive relationships between features at different scales. The generation module is used to process the ground object data to be classified using the trained ground object detection model. After capturing global dependencies through the visual selective state space block (VSS Block) and performing scale competition through the spatial competition selection submodule (SCS Block), a visualization image of the location and category of the ground object is generated.

[0064] Figure 3 The functions of each module in the embodiments correspond to the contents of their respective method embodiments, and will not be repeated here.

[0065] It should be noted that the arrangement of the modules in a flow layout is only one embodiment of the present invention, and other arrangements may also be used, which are not limited in the present invention.

[0066] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for land cover classification based on the combination of hyperspectral imagery and lidar, characterized in that, Includes the following steps: S1: Acquire hyperspectral images and lidar data and perform preprocessing; S2: Label the land cover categories in the processed hyperspectral image and lidar data, and create a label file; S3: Construct a ground target detection model and train it based on the hyperspectral image, lidar data, and the tag file. The ground target detection model includes a multi-scale spectral-spatial sparse coding module for extracting hyperspectral sparse features, a multi-scale geometric sparse coding module for extracting lidar geometric sparse features, and a competitive sparse selection module for dynamic fusion of multi-scale features. The competitive sparse selection module includes: a visually selective state space block for processing small, medium, and large-scale features to capture long-distance spatial dependencies within the features; and a spatial competitive selection submodule for establishing explicit competitive relationships between features at different scales. S4: The trained ground object detection model is used to process the ground object data to be classified. After capturing global dependence through the visual selective state space block and performing scale competition through the spatial competition selection submodule, the location and category of the ground object are generated.

2. The land cover classification method according to claim 1, characterized in that, The preprocessing in step S1 includes: Hyperspectral data preprocessing: The hyperspectral image data is subjected to radiometric calibration, atmospheric correction, geometric correction and format conversion in sequence; LiDAR data preprocessing: The LiDAR data is sequentially subjected to point cloud filtering, point cloud classification, elevation normalization, rasterization, and format conversion; Overall data preprocessing: The preprocessed lidar data and hyperspectral images are registered with coordinate system one and sub-pixel level.

3. The land cover classification method according to claim 1, characterized in that, The annotation in step S2 includes: annotating the land cover categories in the hyperspectral image and lidar data based on ENVI software to generate a label file with the same spatial resolution as the original data; the hyperspectral image data and lidar data share the same label file.

4. The land cover classification method according to claim 1, characterized in that, The process of processing hyperspectral data by the multi-scale spectral-spatial sparse coding module in step S3 is as follows: First, spectral subspace decomposition is performed through grouped convolution; second, a spectral attention mechanism is used to adaptively select discriminative spectral channels; third, spatial patterns are extracted using depth-separable convolutions at different scales; and finally, mode-specific sparsity constraints are achieved through an embedded learnable soft thresholding layer. The process of processing lidar data by the multi-scale geometric sparse coding module is as follows: First, the data is mapped to the feature space through a projection layer; second, depth-separable directional strip convolutions are designed to extract geometric contour features in the horizontal and vertical directions respectively; and finally, the sparsity of the geometric structure is enhanced through an embedded learnable soft thresholding layer.

5. The land cover classification method according to claim 4, characterized in that, The competitive sparse selection module in step S3 adopts a hierarchical cascade strategy: the first level performs spatial competitive selection of large-scale and medium-scale enhanced features, and the second level performs spatial competitive selection of medium-scale and small-scale enhanced features. Feature fusion is achieved by dynamically suppressing redundant scales and enhancing discriminative scale responses.

6. The land cover classification method according to claim 5, characterized in that: The visual selective state space block is based on a two-dimensional selective scanning mechanism. It unfolds the image block into a sequence along four directions: upper left to lower right, lower right to upper left, upper right to lower left, and lower left to upper right. After feature processing using the state space model, the blocks are merged and feature transformation is completed by combining layer normalization and feedforward neural network. The spatial competition selection submodule receives features at two different scales and fused features. It generates pixel-level competition weights in the channel dimension through convolutional layers and the Softmax function. The original features are weighted using the competition weights, and learnable global fusion weights are generated through global average pooling and multilayer perceptron to balance the dynamic relationship between different competition groups.

7. The method for classifying land features according to claim 6, characterized in that, In step S4, target detection is achieved using the final feature representation output by the competitive sparse selection module. The final feature representation is obtained by weighted summation of the outputs of two competing levels through learnable fusion weights. The final feature representation is then input into a classifier to generate the predicted category of the ground target.

8. A land cover classification device based on the combination of hyperspectral imagery and lidar, characterized in that, The land cover classification method as described in any one of claims 1-7 further includes: The acquisition module is used to acquire the hyperspectral image data and the lidar data and perform multi-scale block preprocessing. The annotation module is used to annotate the land cover categories in the processed hyperspectral images and lidar data, and create label files; A construction module is used to build a ground target detection model and train it based on the data and labels. The ground target detection model includes a multi-scale spectral-spatial sparse coding module for extracting hyperspectral sparse features, a multi-scale geometric structure sparse coding module for extracting lidar geometric sparse features, and a competitive sparse selection module for dynamic fusion of multi-scale features. The competitive sparse selection module includes: a visually selective state space block, used to process small, medium, and large-scale features separately to capture long-distance spatial dependencies within the features; and a spatial competitive selection submodule, used to establish explicit competitive relationships between features at different scales. The generation module is used to process the data to be classified using the trained model. After capturing global dependencies through the visual selective state space block and performing scale competition through the spatial competition selection submodule, it outputs the location and category of ground objects.

Images