LiDAR point cloud classification method based on double-flow interaction and multi-scale mixed attention

Abstract

The application provides a LiDAR point cloud classification method based on double-flow interaction and multi-scale mixed attention, relates to the technical field of hyperspectral laser radar point cloud processing, and comprises the following steps: spatial geometric features and spectral features are extracted in parallel through a double-flow interaction encoding module; a bidirectional guiding mechanism is used to realize cross-modal feature complementation; a multi-scale spectral-geometric mixed attention module is designed; spectral channel attention and multi-level K-neighbor-based geometric spatial pyramid attention are used to strengthen the global semantics and key geometric details of deep features; a semantic-spectral contrast loss function is introduced to improve the discrimination ability of the model for homogenous and heterogeneous objects by mining difficult example samples; and finally, the hyperspectral LiDAR point cloud is classified by using the trained model. The application can effectively suppress spectral noise interference, enhance the classification precision of the boundaries of ground objects, and is suitable for the classification of hyperspectral LiDAR point clouds in complex remote sensing scenes.

Description

Technical Field

[0001] This application relates to the field of hyperspectral lidar point cloud processing technology, and in particular to a LiDAR point cloud classification method based on dual-stream interaction and multi-scale hybrid attention. Background Technology

[0002] With the rapid development of 3D scanning technology and remote sensing methods, hyperspectral lidar systems can now simultaneously acquire high-precision 3D spatial coordinates and multi-band spectral information of targets. This data format, which integrates spatial geometry and spectral material properties, provides a rich information foundation for a refined understanding of complex scenes. However, existing point cloud semantic classification methods still have significant limitations when processing such data: On the one hand, most existing algorithms simply treat spectral information as an additional input channel besides geometric coordinates, directly concatenating it into the neural network. This ignores the physical complementarity and nonlinear coupling between geometry and spectral material, resulting in the network struggling to extract discriminative features in complex boundary regions where the geometry is similar but the material is different, or the material is similar but the geometry is vastly different. On the other hand, in the feature extraction process of deep neural networks, as the number of network layers increases, the feature dimension increases but the spatial resolution decreases sharply. Existing network architectures lack effective attention mechanisms for the characteristics of disordered point clouds at bottleneck layers, making it impossible to accurately filter out key virtual spectral channels and geometric key points from high-dimensional features lacking spatial details, thus causing a loss of segmentation accuracy for small objects or fine edges.

[0003] Furthermore, existing supervised training systems primarily rely on cross-entropy loss functions based on classification labels. This single supervisory signal struggles to handle complex lighting variations, shadow occlusion, and metaspectral interference in real-world scenarios. In practical applications, the spectral feature distributions of the same type of object vary significantly across different regions, a phenomenon known as "different spectra for the same object"; conversely, different types of objects may possess highly similar spectral features, a phenomenon known as "different objects with the same spectrum." Traditional classification loss alone cannot force the network to uncover essential semantic features beyond the apparent spectral values, resulting in poor generalization ability and widespread misclassification when faced with uneven lighting or camouflaged interference. Summary of the Invention

[0004] The purpose of this invention is to address the problems of rigid feature fusion, insufficient deep semantic capture capability, and poor model robustness caused by a single supervision signal in existing hyperspectral LiDAR point cloud classification methods, and to provide a LiDAR point cloud classification method based on dual-stream interaction and multi-scale hybrid attention.

[0005] The above-mentioned objective of this application is achieved through the following technical solution: S1: Acquire hyperspectral LiDAR point cloud data and extract spatial and spectral features; S2: Construct a hierarchical feature extraction network containing encoder paths, bottleneck layers, and decoder paths, and input spatial and spectral features into the encoder; S3: In each level of the encoder path, spatial flow features and spectral flow features are extracted in parallel through the dual-stream interaction module. The local consistency of spatial features is used to guide the aggregation of spectral features, and the boundary gating coefficients are generated using spectral differences to correct geometric features, thus obtaining spatial-spectral features. S4: Set up a multi-scale spectral-geometric hybrid attention module in the bottleneck layer to enhance the spatial-spectral features of the encoder output and obtain high-dimensional features; S5: Each layer of the decoder upsamples the high-dimensional features, fuses the shallow features through skip connections, and finally outputs the point-by-point semantic classification result.

[0006] Optionally, step S1 includes: Acquire hyperspectral LiDAR point cloud data, normalize the spatial coordinates and spectral data respectively, and extract spatial features and spectral features; Spatial features of normalized hyperspectral LiDAR point cloud data are extracted using sparse convolution. Spectral feature extraction of hyperspectral LiDAR point cloud data employs a fully connected layer to map to a high dimension.

[0007] Optionally, step S2 includes: A hierarchical feature extraction network is constructed, which includes an encoder path, a bottleneck layer, and a decoder path. The extracted spatial features and spectral features are used as inputs to the encoder path.

[0008] Optionally, step S3 includes: The dual-stream interaction module specifically includes: Construct a local K-nearest neighbor graph and extract geometric and spectral features; For spatial feature extraction, a local graph is constructed using K-nearest neighbors, and geometric features are extracted using EdgeConv.

[0009] in, For point The neighborhood, Indicates a splicing operation; and They represent the first The and the first Spatial feature vectors of points; Represents a multilayer perceptron; This represents the spatial features after feature extraction; In terms of spectral feature extraction, 1D convolution is used to extract spectral features:

[0010] in, This represents the spectral features after feature extraction. Represents 1D convolution; This represents the input spectral feature vector; The cosine similarity matrix of the geometric features is calculated and used as weights to perform weighted smoothing on the spectral features of the neighborhood, generating geometrically enhanced spectral features. Calculate the Euclidean distance difference of spectral features to generate boundary gating coefficients; By utilizing boundary gating coefficients to suppress the aggregation of geometric features across spectral boundaries, spectrally enhanced geometric features are generated. Spatial-spectral features are generated through geometrically enhanced spectral features and spectrally enhanced geometric features.

[0011] Optionally, step S4 includes: The multi-scale spectral-geometric hybrid attention module includes: Spectral channel attention branch: Channel weights are generated through global average pooling and global max pooling to recalibrate feature channels; Geometric spatial pyramid branch: Construct a multi-level K-nearest neighbor graph, aggregate multi-scale neighborhood features, and generate a spatial attention graph; Fusion Branch: The outputs of the two branches are weighted and merged, and then connected by residuals.

[0012] Optionally, step S5 includes: Construct a joint loss function to train a hierarchical feature extraction network; The joint loss function includes cross-entropy loss and semantic-spectral contrast loss; The semantic-spectral contrast loss is calculated in the following way: In the feature space, a set of positive and negative samples is constructed based on the true labels; The top K samples with the largest spectral Euclidean distance among similar samples are selected as the difficult positive samples; The top K samples with the smallest spectral Euclidean distance among the outliers are selected as the hardest negative samples; Based on the InfoNCE loss function, the comparative loss is calculated only for hard positive samples and hard negative samples, which brings the feature distance of similar samples closer and pushes the feature distance of dissimilar samples further apart.

[0013] Optionally, step S6 includes: Each decoder layer upsamples and processes the high-dimensional features of the bottleneck layer, and makes skip connections with the corresponding coding layer features. After being fused by a multilayer perceptron, the features are input to the next decoding layer. The classification head maps and outputs point-by-point semantic classification prediction results. Upsampling uses inverse distance weighted interpolation, where interpolated features are skipped to the corresponding coding layer features, and then fused through a multilayer perceptron before being input to the next decoding layer.

[0014] An electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform a LiDAR point cloud classification method based on dual-stream interaction and multi-scale hybrid attention.

[0015] A computer-readable storage medium storing instructions that, when executed, perform a LiDAR point cloud classification method based on dual-stream interaction and multi-scale hybrid attention.

[0016] The beneficial effects of the technical solution provided in this application are: This invention proposes a hyperspectral LiDAR point cloud classification method based on spatial-spectral self-supervised pre-training. By constructing a two-stream interactive coding mechanism, this invention effectively suppresses noise interference in spectral data by utilizing the local consistency of geometric features. Simultaneously, it utilizes the differences in spectral features to generate gating coefficients to sharpen geometric boundaries, completely solving the problems of abrupt feature fusion and blurred object edge segmentation caused by traditional simple stitching. This achieves deep complementary enhancement of geometric and spectral modes. Furthermore, a multi-scale spectral-geometric hybrid attention module is introduced into the deep layers of the network, innovatively employing a multi-level K-nearest neighbor aggregation strategy to replace the traditional... Fixed-scale convolution can adapt to the disordered distribution characteristics of point clouds, and strengthen the feature representation of key geometric parts and high-dimensional virtual spectral channels at multiple levels from microscopic details to macroscopic context, effectively compensating for the loss of spatial detail information during downsampling. In addition, by combining the training strategy of semantic-spectral contrast loss, the network is forced to learn essential semantic features that are robust to changes in illumination and shadow occlusion by selectively mining and optimizing hard examples in the feature space, overcoming the interference of "same object, different spectrum" and "same spectrum, different object" phenomena, thereby significantly improving the generalization ability and classification accuracy of the classification model in complex remote sensing scenarios. Attached Figure Description

[0017] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of an embodiment of this application; Figure 2This is a visualization of the Harbor of Tobermory Dataset hyperspectral LiDAR point cloud remote sensing dataset used in the embodiments of this application; Figure 3 This is a visualization of the hyperspectral LiDAR point cloud classification in the embodiments of this application; Figure 4 This is a schematic diagram of the electronic device structure in the embodiments of this application. Detailed Implementation

[0018] To provide a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0019] The embodiments of this application provide a LiDAR point cloud classification method based on dual-stream interaction and multi-scale hybrid attention.

[0020] Please refer to Figure 1 , Figure 1 This is a flowchart of a LiDAR point cloud classification method based on two-stream interaction and multi-scale hybrid attention in an embodiment of this application, including: S1: Acquire hyperspectral LiDAR point cloud data and extract spatial and spectral features; S2: Construct a hierarchical feature extraction network containing encoder paths, bottleneck layers, and decoder paths, and input spatial and spectral features into the encoder; S3: In each level of the encoder path, spatial flow features and spectral flow features are extracted in parallel through the dual-stream interaction module. The local consistency of spatial features is used to guide the aggregation of spectral features, and the boundary gating coefficients are generated using spectral differences to correct geometric features, thus obtaining spatial-spectral features. S4: Set up a multi-scale spectral-geometric hybrid attention module in the bottleneck layer to enhance the spatial-spectral features of the encoder output and obtain high-dimensional features; S5: Each layer of the decoder upsamples the high-dimensional features, fuses the shallow features through skip connections, and finally outputs the point-by-point semantic classification result.

[0021] This application constructs a deep neural network for hyperspectral LiDAR point cloud classification in remote sensing scenarios by adopting the above-mentioned technical solution. In the encoder stage, a cascaded dual-stream interaction module is innovatively designed to extract geometric and spectral features in parallel. Through a bidirectional guidance mechanism, the smoothing weights are calculated using the local consistency of geometric features to eliminate noise interference in spectral data. At the same time, the boundary gating coefficients are generated using the local differences of spectral features to suppress excessive smoothing of geometric features at object edges, thereby achieving complementary enhancement of geometric and spectral features. At the deep bottleneck of the network, this invention introduces a multi-scale spectral-geometric hybrid attention module. This module automatically identifies and recalibrates high-dimensional virtual spectral channels that are important for the classification task through the spectral channel attention branch. At the same time, through the geometric spatial pyramid branch built based on multi-level K-nearest neighbors, geometric context information from different scales from micro to macro is aggregated to generate a spatial attention map to strengthen the feature representation of key geometric parts. Finally, the decoder restores the feature resolution and outputs the prediction results through inverse distance weighted interpolation and skip connections.

[0022] As one embodiment, to address the challenge of complex spectral noise, this invention constructs a joint loss function during the model training phase, incorporating cross-entropy loss and semantic-spectral contrastive loss. The semantic-spectral contrastive loss is based on a hard example mining strategy in the feature space, specifically selecting hard positive samples that are of the same class as the anchor point but have the largest spectral distance, and hard negative samples that are of different class than the anchor point but have the smallest spectral distance. The InfoNCE loss form forces the network to narrow the feature distance of hard positive samples and widen the feature distance of hard negative samples. Through this technical solution, this invention effectively utilizes geometric structure to constrain spectral noise, sharpens geometric boundaries using spectral differences, and significantly improves the accuracy and robustness of hyperspectral LiDAR point cloud classification through a multi-scale hybrid attention mechanism and contrastive learning strategy.

[0023] Step S1 includes: Acquire hyperspectral LiDAR point cloud data, normalize the spatial coordinates and spectral data respectively, and extract spatial features and spectral features; Spatial features of normalized hyperspectral LiDAR point cloud data are extracted using sparse convolution. Spectral feature extraction of hyperspectral LiDAR point cloud data employs a fully connected layer to map to a high dimension.

[0024] As one example, reading the input hyperspectral LiDAR point cloud , where each point Includes spatial coordinates and spectral information The spatial coordinates were normalized to within a unit sphere, and the spectral data were normalized to... For the normalized point cloud spatial features, sparse convolution is used to extract high-dimensional features; for the normalized point cloud spectral features, a fully connected layer is used to map to high dimensions and extract high-dimensional features.

[0025] Step S2 includes: A hierarchical feature extraction network is constructed, which includes an encoder path, a bottleneck layer, and a decoder path. The extracted spatial features and spectral features are used as inputs to the encoder path.

[0026] Step S3 includes: The dual-stream interaction module specifically includes: Construct a local K-nearest neighbor graph and extract geometric and spectral features; For spatial feature extraction, a local graph is constructed using K-nearest neighbors, and geometric features are extracted using EdgeConv.

[0027] in, For point The neighborhood, Indicates a splicing operation; and They represent the first The and the first Spatial feature vectors of points; This represents a multilayer perceptron; This represents the spatial features after feature extraction; In terms of spectral feature extraction, 1D convolution is used to extract spectral features:

[0028] in, This represents the spectral features after feature extraction. Represents 1D convolution; This represents the input spectral feature vector; The cosine similarity matrix of the geometric features is calculated and used as weights to perform weighted smoothing on the spectral features of the neighborhood, generating geometrically enhanced spectral features. Calculate the Euclidean distance difference of spectral features to generate boundary gating coefficients; By utilizing boundary gating coefficients to suppress the aggregation of geometric features across spectral boundaries, spectrally enhanced geometric features are generated. Spatial-spectral features are generated through geometrically enhanced spectral features and spectrally enhanced geometric features.

[0029] As one embodiment, a dual-stream interaction module is constructed to enhance spatial-spectral features. The dual-stream interaction module calculates the local similarity matrix of geometric features and uses it as weight to perform weighted smoothing of spectral features in the neighborhood to generate geometrically enhanced spectral features. At the same time, it calculates the local Euclidean distance difference of spectral features to generate boundary gating coefficients. The gating coefficients are used to suppress the aggregation of geometric features across spectral boundaries to generate spectrally enhanced geometric features.

[0030] Step S4 includes: The multi-scale spectral-geometric hybrid attention module includes: Spectral channel attention branch: Channel weights are generated through global average pooling and global max pooling to recalibrate feature channels; Geometric spatial pyramid branch: Construct a multi-level K-nearest neighbor graph, aggregate multi-scale neighborhood features, and generate a spatial attention graph; Fusion Branch: The outputs of the two branches are weighted and merged, and then connected by residuals.

[0031] As one example, the spectral channel attention branch compresses the spatial dimension to 1 by performing global average pooling and global max pooling on the input features, and generates a spectral channel weight vector by learning the dependencies between channels through a shared multilayer perceptron, which is used to recalibrate the feature channels. As one example, the geometric spatial pyramid branch constructs a multi-level K-nearest neighbor graph for each center point, aggregating neighborhood features at different scales; after splicing the aggregated features at different scales, they are fused through a convolutional layer to generate a spatial attention map, which is used to weight and enhance key geometric points; As one example, the output features of the spectral channel attention branch and the geometric space pyramid branch are weighted and fused together, and then residually connected.

[0032] Step S5 includes: Construct a joint loss function to train a hierarchical feature extraction network; The joint loss function includes cross-entropy loss and semantic-spectral contrast loss; The semantic-spectral contrast loss is calculated in the following way: In the feature space, a set of positive and negative samples is constructed based on the true labels; The top K samples with the largest spectral Euclidean distance among similar samples are selected as the difficult positive samples; The top K samples with the smallest spectral Euclidean distance among the outliers are selected as the hardest negative samples; Based on the InfoNCE loss function, the comparative loss is calculated only for hard positive samples and hard negative samples, which brings the feature distance of similar samples closer and pushes the feature distance of dissimilar samples further apart.

[0033] As one embodiment, in the feature space, for any anchor point, a positive sample set and a negative sample set are constructed based on the real labels; the top K samples that are of the same type as the anchor point but have the largest Euclidean distance from the original spectrum are defined as difficult positive samples; the top K samples that are of different type as the anchor point but have the smallest Euclidean distance from the original spectrum are defined as difficult negative samples; based on the InfoNCE loss function, the contrast loss is calculated only for the difficult positive samples and difficult negative samples, forcing the network to bring the feature distance of the difficult positive samples closer and push the feature distance of the difficult negative samples further away.

[0034] Step S6 includes: Each decoder layer upsamples and processes the high-dimensional features of the bottleneck layer, and makes skip connections with the corresponding coding layer features. After being fused by a multilayer perceptron, the features are input to the next decoding layer. The classification head maps and outputs point-by-point semantic classification prediction results. Upsampling uses inverse distance weighted interpolation, where interpolated features are skipped to the corresponding coding layer features, and then fused through a multilayer perceptron before being input to the next decoding layer.

[0035] In one embodiment, experimental data: Harbor of Tobermory dataset (HT): such as Figure 2 As shown, this dataset is a real hyperspectral LiDAR point cloud remote sensing dataset. HT contains 7,181,982 spatial points collected using the Optech Titan three-channel LiDAR system, which acquires data in three spectral bands (532nm, 1064nm, and 1550nm). The dataset is manually labeled into nine land cover categories: shrubs, buildings, vehicles, grasslands, power lines, roads, boats, trees, and water.

[0036] Experimental setup: The example network of this invention is based on PyTorch 2.1.1 (compile-time supports CUDA 11.8) and hardware acceleration is enabled through the NVIDIA 535.98 driver. We uniformly selected the following evaluation metrics: overall accuracy (OA), average F1 score (Avg. F1), and average intersection-over-union ratio (mIoU).

[0037] Step S1: Data Preprocessing and Feature Embedding Let the input hyperspectral LiDAR point cloud set be... , where each point Includes spatial coordinates and spectral information The 3D spatial features of the point cloud are all normalized using min-max normalization: The three-band spectral features of the point cloud are scaled to the [0,1] interval and then normalized. Regarding spatial features Sparse convolution is used to extract features; for spectral features... A fully connected layer is used to map to a higher dimension:

[0038]

[0039] in, Represents sparse convolution. Indicates a fully connected layer. and Representing the high-dimensional spatial features and spectral features after high-dimensional mapping, respectively, with an output dimension of 1. , indicating the embedding dimension.

[0040] Step S2: Construct the two-stream interactive encoding module, the encoder's first... Hierarchy ( The number of hyperspectral LiDAR point cloud sample points is [number]. The input space features are Spectral characteristics are First, spatial and spectral features are extracted. For spatial feature extraction, a local graph is constructed using K-Nearest Neighbors (KNN), and EdgeConv is used to extract geometric features. ,in, For point The neighborhood, This indicates a splicing operation. For spectral feature extraction, 1D convolution is used to extract spectral features: .in, and These represent the spatial and spectral features after feature extraction, respectively. Geometric-guided spectral aggregation and spectral-guided geometric correction operations are performed based on the extracted spatial and spectral features. For each point, a local graph is constructed using K-nearest neighbors (KNN), and geometric-guided spectral aggregation is performed. The cosine similarity of the geometric features is calculated as a smoothing weight to weight the neighborhood spectral features.

[0041]

[0042] in, Here is the weight normalization function. This is the spatial similarity metric matrix between the center point and its neighboring points. These are learnable scaling factors. This step utilizes geometric consistency to smooth spectral noise. For each point, a local graph is constructed based on K-nearest neighbors (KNN), a spectral-guided geometric correction operation is performed, the Euclidean distance of the spectral features is calculated, and boundary gating coefficients are generated. The specific process is as follows:

[0043]

[0044]

[0045] in, It is the Sigmoid activation function. This is the spectral difference measurement matrix. The gating coefficients for each neighboring point relative to the center point are: It is a fully connected layer. When there are spectral differences... When it is large (such as the boundary of an object), that is This breaks down the aggregation of geometric features, preserving boundary clarity. Finally, the interacting features... and The layers are spliced ​​together and then skipped to connect with the subsequent decoding layers. and As input to the next coding layer.

[0046] Step S3: After multiple layers of encoding and feature extraction, at the bottleneck layer, let the input features be... ,in, Include and These represent the feature extraction results guided by spatial and spectral information under the encoder path, respectively. A spectral channel attention branch and a geometric spatial pyramid branch are designed based on the two types of feature extraction results. The spectral channel attention branch recalibrates the channel using global statistics, as detailed below:

[0047]

[0048]

[0049] in, This indicates multiplication by channel. For the calculated spectral channel attention weights, is a non-linear activation function. In the geometric space pyramid branch, for each point, a set of K nearest neighbors at three scales is constructed: For each scale Perform feature aggregation: The spatial attention map is generated by fusing multi-scale features, and the specific process is as follows:

[0050]

[0051] in, This indicates point-by-point multiplication. To calculate the spatial channel attention weights, It is a non-linear activation function. This represents a convolution operation. Finally, the geometric-spectral dual-branch output data are fused to obtain spatial-spectral features. : This serves as the input for subsequent decoder paths.

[0052] Step S4: For the high-dimensional features of the bottleneck layer, inverse distance weighted interpolation is used for upsampling. The features obtained after sampling are then connected with the output data of the corresponding layer of the geometric-spectral dual branch by a jump connection.

[0053] Step S5: Based on the joint loss function for hard example mining, the total loss function is defined as follows:

[0054] in, For standard cross-entropy loss, , For the predicted value of the classification head, It is true. The predicted number of categories is 9. For semantic-spectral contrast loss, The balancing coefficient is used. A semantic-spectral contrastive loss is applied by constructing a set of difficult examples. The calculation involves first defining the spectral distance and setting the anchor point. The original spectral vector is Candidate points The spectral vector is Spectral distance is defined as A set of difficult examples is constructed based on the defined spectral distance. This set includes both a set of difficult positive examples and a set of difficult negative examples. Select from a sample set with the same labels The largest front These points serve as the difficult-to-correct sample set: In relation to Select from sample sets with different labels The smallest front 1 point as the hard-to-bear sample set: .set up The bottleneck layer of the network outputs a feature vector that has been normalized by the projection head. Finally, the contrastive loss is calculated. The specific process is as follows:

[0055] in, The temperature coefficient is 0.07 in this embodiment. This formula is optimized for similar points with large spectral differences and dissimilar points with similar spectra, forcing the network to learn essential semantic features beyond the spectral appearance.

[0056] Step S6: Fuse the enhanced spatial-spectral features from the bottleneck layer, restore feature resolution through inverse distance weighted interpolation upsampling, and fuse shallow features from the encoder path with deep features from the decoder path using skip connections. Finally, output point-by-point semantic classification prediction results after mapping by the classification head. The specific process is as follows: For each decoding stage in the encoder-decoder architecture (let's assume the current stage is the...),... The next layer is The layer takes the bottleneck layer features enhanced by multi-scale hybrid attention or the output features of the previous decoding layer as input. Because the number of point clouds decreases layer by layer during the encoder process (…), the input is either the bottleneck layer features after multi-scale hybrid attention enhancement or the output features of the previous decoding layer. The inverse distance weighted interpolation method needs to be used to reduce the number of feature points from... Restore to For the first arbitrary interpolation points in the layer In the Search for it in the layer nearest neighbor ( ), calculate the interpolated features:

[0057]

[0058] in, Represents the spatial Euclidean distance. The distance is a power of 1. To prevent the minimum value of division by zero, For the first The input features of the layer. The interpolated features obtained from upsampling. Shallow spatial-spectral features extracted from the corresponding layer in the encoder path Skip connections are performed. A concatenation operation is used to fuse deep semantics and shallow details, and then feature integration is performed through a multilayer perceptron to obtain the output features of this layer of the decoder. The specific process is as follows:

[0059] in, This represents a stitching operation along the feature channel dimension. This process is repeated until the feature resolution is restored to the original point cloud point count. Once the features are restored to their original resolution, the final point-by-point feature representation is obtained. This data is then fed into a classification head, which consists of fully connected layers and a softmax activation function, mapping the feature dimensions to a class probability distribution.

[0060]

[0061] in, For the prediction probability matrix, The total number of land cover categories (in this embodiment) ), For the first The final semantic category label for each point. The final generated prediction results will be used for subsequent accuracy evaluation and visualization (e.g., Figure 3 (As shown).

[0062] Experimental results: Following the steps above, experiments were conducted on the Harbor of Tobermory dataset to verify the predictions. The experimental results are shown in Table 1, where the ground truth labels and the predicted labels for each method are visualized. Figure 3 .

[0063] Table 1. Prediction performance of the present invention and the comparison method on real hyperspectral LiDAR point cloud datasets.

[0064] As shown in Table 1, the optimal values ​​for each indicator are highlighted in bold. The proposed method significantly improves classification performance on the dataset. Compared to the suboptimal method, the overall accuracy is improved by 9.2%, the average F1 score is improved by 10.6%, and the average intersection-union ratio is significantly improved by 14.5%. This indicates that the proposed method can better utilize the two-stream interaction mechanism to capture the complex spatial-spectral complementary features of large-scale hyperspectral LiDAR point cloud data than existing techniques. Figure 3 The visualizations compare the prediction results of our method with those of other classification methods, more intuitively demonstrating the advantages of our method. For example... Figure 3 As shown, the proposed method not only exhibits superior regional consistency for large-scale categories such as buildings, but also demonstrates more accurate boundary delineation for fine-grained features (vehicles, power lines). This proves that the proposed method is more accurate in observing real-world scenes compared to existing techniques.

[0065] This application also discloses an electronic device. (See reference...) Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0066] The communication bus 502 is used to enable communication between these components.

[0067] The user interface 503 may include a display screen, and optionally, the user interface 503 may also include a standard wired interface or a wireless interface.

[0068] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0069] This application also discloses a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the aforementioned LiDAR point cloud classification method based on dual-stream interaction and multi-scale hybrid attention.

[0070] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure.

[0071] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A LiDAR point cloud classification method based on two-stream interaction and multi-scale hybrid attention, characterized in that, The method includes the following steps: S1: Acquire hyperspectral LiDAR point cloud data and extract spatial and spectral features; S2: Construct a hierarchical feature extraction network containing encoder paths, bottleneck layers, and decoder paths, and input spatial and spectral features into the encoder; S3: In each level of the encoder path, spatial flow features and spectral flow features are extracted in parallel through the dual-stream interaction module. The local consistency of spatial features is used to guide the aggregation of spectral features, and the boundary gating coefficients are generated using spectral differences to correct geometric features, thus obtaining spatial-spectral features. S4: Set up a multi-scale spectral-geometric hybrid attention module in the bottleneck layer to enhance the spatial-spectral features of the encoder output and obtain high-dimensional features; S5: Each layer of the decoder upsamples the high-dimensional features, fuses the shallow features through skip connections, and finally outputs the point-by-point semantic classification result.

2. The LiDAR point cloud classification method based on two-stream interaction and multi-scale hybrid attention as described in claim 1, characterized in that, Step S1 includes: Acquire hyperspectral LiDAR point cloud data, normalize the spatial coordinates and spectral data respectively, and extract spatial features and spectral features; Spatial features of normalized hyperspectral LiDAR point cloud data are extracted using sparse convolution. Spectral feature extraction of hyperspectral LiDAR point cloud data employs a fully connected layer to map to a high dimension.

3. The LiDAR point cloud classification method based on two-stream interaction and multi-scale hybrid attention as described in claim 1, characterized in that, Step S2 includes: A hierarchical feature extraction network is constructed, which includes an encoder path, a bottleneck layer, and a decoder path. The extracted spatial features and spectral features are used as inputs to the encoder path.

4. The LiDAR point cloud classification method based on two-stream interaction and multi-scale hybrid attention as described in claim 1, characterized in that, Step S3 includes: The dual-stream interaction module specifically includes: Construct a local K-nearest neighbor graph and extract geometric and spectral features; For spatial feature extraction, a local graph is constructed using K-nearest neighbors, and geometric features are extracted using EdgeConv. in, For point The neighborhood, Indicates a splicing operation; and They represent the first The and the first Spatial feature vectors of points; Represents a multilayer perceptron; This represents the spatial features after feature extraction; In terms of spectral feature extraction, 1D convolution is used to extract spectral features: in, This represents the spectral features after feature extraction. Represents 1D convolution; This represents the input spectral feature vector; The cosine similarity matrix of the geometric features is calculated and used as weights to perform weighted smoothing on the spectral features of the neighborhood, generating geometrically enhanced spectral features. Calculate the Euclidean distance difference of spectral features to generate boundary gating coefficients; By utilizing boundary gating coefficients to suppress the aggregation of geometric features across spectral boundaries, spectrally enhanced geometric features are generated. Spatial-spectral features are generated through geometrically enhanced spectral features and spectrally enhanced geometric features.

5. The LiDAR point cloud classification method based on two-stream interaction and multi-scale hybrid attention as described in claim 1, characterized in that, Step S4 includes: The multi-scale spectral-geometric hybrid attention module includes: Spectral channel attention branch: Channel weights are generated through global average pooling and global max pooling to recalibrate feature channels; Geometric spatial pyramid branch: Construct a multi-level K-nearest neighbor graph, aggregate multi-scale neighborhood features, and generate a spatial attention graph; Fusion Branch: The outputs of the two branches are weighted and merged, and then connected by residuals.

6. The LiDAR point cloud classification method based on two-stream interaction and multi-scale hybrid attention as described in claim 1, characterized in that, Step S5 includes: Construct a joint loss function to train a hierarchical feature extraction network; The joint loss function includes cross-entropy loss and semantic-spectral contrast loss; The semantic-spectral contrast loss is calculated in the following way: In the feature space, a set of positive and negative samples is constructed based on the true labels; The top K samples with the largest spectral Euclidean distance among similar samples are selected as the difficult positive samples; The top K samples with the smallest spectral Euclidean distance among the outliers are selected as the hardest negative samples; Based on the InfoNCE loss function, the comparative loss is calculated only for hard positive samples and hard negative samples, which brings the feature distance of similar samples closer and pushes the feature distance of dissimilar samples further apart.

7. The LiDAR point cloud classification method based on two-stream interaction and multi-scale hybrid attention as described in claim 1, characterized in that, Step S6 includes: Each decoder layer upsamples and processes the high-dimensional features of the bottleneck layer, and makes skip connections with the corresponding coding layer features. After being fused by a multilayer perceptron, the features are input to the next decoding layer. The classification head maps and outputs point-by-point semantic classification prediction results. Upsampling uses inverse distance weighted interpolation, where interpolated features are skipped to the corresponding coding layer features, and then fused through a multilayer perceptron before being input to the next decoding layer.

8. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform the LiDAR point cloud classification method based on dual-stream interaction and multi-scale hybrid attention as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform the LiDAR point cloud classification method based on dual-stream interaction and multi-scale hybrid attention as described in any one of claims 1-7.

Images