Road scene semantic segmentation method based on laser radar point cloud data

Abstract

The invention discloses a road scene semantic segmentation method based on laser radar point cloud data, and relates to the technical field of engineering construction, and the method comprises the following steps: S1, carrying out the preprocessing of an original point cloud of a laser radar, and representing the point cloud data through employing a sparse tensor; s2, a lightweight road point cloud semantic segmentation network RWAFormer based on a Transform is designed, and the RWAFormer of the road point cloud semantic segmentation network RWAFormer based on the Transform is designed; s3, pre-training and parameter optimization are carried out on the lightweight road point cloud semantic segmentation network RWAFormer based on the Transform constructed in the step S2, and the parameter optimization is carried out on the lightweight road point cloud semantic segmentation network RWAFormer based on the Transform constructed in the step S2; s4, inputting the preprocessed test set data set into the trained road point cloud semantic segmentation method network in the S3, and outputting a final semantic category; according to the method, a laser radar road point cloud data set in a traffic scene is researched, and a new U-Net architecture model RWAFormer is provided to solve the semantic segmentation problem of large-scale laser radar road point cloud by using the characteristics of laser radar road point cloud data.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a road scene semantic segmentation method based on laser radar point cloud data. Background Art

[0002] In recent years, with the rapid development of 3D sensing technology, the quality of 3D point cloud data has been continuously improved. Since point cloud data can well preserve information in three-dimensional space, point cloud-based 3D computer vision tasks have achieved good results, further promoting its development in various 3D application scenarios. For example, a large number of road point clouds can be used in autonomous driving, intelligent transportation, and urban planning. In road scenes, road point cloud semantic segmentation is the basis for perceiving and understanding road scenes, aiming to assign specific semantic categories to each point in the road point cloud. For example, in autonomous driving, it is necessary to identify and perceive targets such as pedestrians, vehicles, and traffic lights to provide information for subsequent decision-making. However, the road point cloud based on LiDAR has the characteristics of large scale and uneven density, and it is challenging to achieve accurate and efficient semantic segmentation.

[0003] In recent years, deep learning technology has been widely used in road point cloud segmentation in traffic scenes. PointNet proposed in the prior art uses a single-branch structure to deepen the network and generates a score for each point by splicing global features and local features. Since PointNet++ and D-PointNet++ use the farthest point sampling (FPS) to aggregate local features of point clouds, adding a local feature aggregation module to extract local features has become a mainstream trend. FPS can ensure that the centroid is evenly distributed in the point cloud and aggregate local features in the point cloud. DGCNN uses k-nearest neighbors (k-NN) to construct a local neighborhood graph and updates the dynamic graph to make the receptive field as large as possible, close to the diameter of the point cloud. PointMLP uses a geometric affine module to extract local point features before and after the aggregation operation.

[0004] In addition, Transformer-based architectures have achieved remarkable success in visual tasks, thanks to the spatial modeling method based on the dot product self-attention mechanism. In order to further utilize the spatial characteristics of features in higher-order interactions, HorNet adopts a recursive structure called gnConv. Although the various networks proposed in the above studies have shown good results in point cloud semantic segmentation, they do not fully consider the characteristics of road point cloud data, such as natural disorder, large data volume, and irregular distribution of scene information. Summary of the invention

[0005] This paper studies the LiDAR road point cloud dataset in traffic scenarios, and proposes a new U-Net architecture model based on the characteristics of LiDAR road point cloud data to solve the semantic segmentation problem of large-scale LiDAR road point clouds.

[0006] In view of the above problems, the present invention focuses on developing a road scene semantic segmentation method based on LiDAR point cloud data. The overall network structure of the model is as follows: Figure 1 As shown in the figure, in general, we combine the vector attention mechanism with the U-Net encoder-decoder framework, where the U-net structure contains 5 encoders and 5 decoders. The encoder consists of a downsampling module, a sparse tensor feature extraction module (STFE) and a radial window attention module (RWA) proposed in this paper to capture features, and the decoder consists of an upsampling module, STFE and RWA to map features. The RWA is based on radial window self-attention, which can obtain effective feature information of distant points from dense point areas, overcome the problem of disconnection of sparse distant points, and increase the effective receptive field.

[0007] Specifically, during the preprocessing of the original LiDAR point cloud: Since road scenes usually contain a large number of scattered points, and in the road point cloud segmentation task, efficient processing is required to meet the needs of practical applications. The sparse tensor feature extraction module processes directly on the original point cloud through continuous convolution operations instead of converting the point cloud into a dense voxel grid, thereby retaining the sparsity of the point cloud and reducing the consumption of computing resources. Its network architecture design is simple and efficient.

[0008] The input point cloud data is first encoded by a sparse tensor, encoding each point into a high-dimensional vector while maintaining its spatial position information. Subsequently, these encoded point cloud data are extracted through 8 consecutive Minkowski convolutional layers, which can effectively extract local and global features while maintaining data sparsity. Between these convolutional layers, there are activation functions and regularization layers to enhance the nonlinear fitting ability of the network and prevent overfitting.

[0009] Embedding Radial Window Attention Module RWA: Since the distribution of LiDAR point clouds is sparse, the lack of neighboring points near sparse distant points leads to disconnected feature information, which hinders the expansion of the receptive field and results in poor segmentation of distant points. The RWA module proposed in this paper can effectively capture the global and local information in the point cloud and directly aggregate the long-distance feature information into a single operator to adapt to the sparse distribution of point clouds.

[0010] The input is an N×C local feature matrix extracted by the sparse tensor feature extraction module. First, the input local features use a radial window mechanism to dynamically select the size of the local neighborhood according to the radius of the center point.

[0011] Subsequently, spherical convolution is performed on the input features. By designing a convolution kernel that adapts to the sphere, spherical convolution can more accurately extract the local geometric features of the point cloud while maintaining the integrity of the spatial information. Therefore, this convolution method is particularly suitable for processing point cloud data with irregular distribution characteristics.

[0012] After the spherical convolution, RWA applies a multi-head attention mechanism, which enables the model to learn information in multiple different feature subspaces at the same time, which helps capture the complex dependencies between different features. After a series of spherical convolutions and multi-head attention processing, RWA integrates the extracted features together.

[0013] The technical solution of the present invention is: a road scene semantic segmentation method based on laser radar point cloud data, which is special in that it includes the following steps: In RWAFormer, the sparse tensor feature encoding module (STFE) is introduced. Through its unique continuous convolution operation and sparse tensor encoding mechanism, efficient feature extraction is achieved while retaining the intrinsic characteristics of the point cloud.

[0014] A radial window attention module (RWA) is designed. Through the radial window attention mechanism, the information of long-distance sparse point clouds can be effectively aggregated to the adjacent dense areas, which significantly improves the segmentation quality of point clouds at long distances. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The road scene semantic segmentation model based on laser radar point cloud data provided by the present invention; Figure 2 A diagram of a sparse tensor feature extraction module provided by the present invention; Figure 3 A diagram of a radial window attention module provided by the present invention; Figure 4 This is a visualization diagram of the segmentation results based on the Semantickitti dataset of the present invention; Figure 5 This is a visualization diagram of the segmentation results based on the Nuscenes dataset of the present invention; Figure 6 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Aiming at the semantic feature extraction of road point clouds, the present invention designs a sparse tensor feature extraction module to capture the key geometric and spatial features in the point cloud and perform feature extraction on the input point cloud data. The sparse tensor feature extraction module processes directly on the original point cloud through continuous convolution operations instead of converting the point cloud into a dense voxel grid, thereby retaining the sparsity of the point cloud and reducing the consumption of computing resources. Figure 2 As shown in Figure 2, the working process of the sparse tensor feature extraction module is as follows: Step 1: Sparse tensor encoding is the first and core step in processing point cloud data in the sparse tensor feature extraction module. It represents each point as a high-dimensional vector and records its spatial position information. This encoding method not only retains the geometric characteristics of the point cloud, but also provides a suitable input format for subsequent continuous convolution operations.

[0019] Step 2: The specific encoding method is as follows: This module uses sparse tensors to represent point cloud data, and represents the point cloud into two parts: coordinate matrix C and feature matrix F.

[0020] (1.1) in represents the coordinates of the point cloud, Indicates which point cloud in the batch it belongs to, N indicates the number of all points in 1 batch, Represents the feature of the i-th point. This representation method can effectively save space and improve computational efficiency.

[0021] Step 3: Another core of the sparse tensor feature extraction module is the continuous convolution layer. Unlike traditional convolution, it operates directly on the original point cloud data and does not rely on any grid structure. The continuous convolution layer uses a differentiable nearest neighbor search method (NNS) to determine other points in the local neighborhood of each point, and then performs Minkowski convolution operations in this local neighborhood. Finally, the local feature map of the input point cloud is obtained. Minkowski convolution not only maintains the inherent sparsity of point cloud data, but also can effectively capture local features.

[0022] Step 4: Therefore, STFE can better preserve the spatial structure of point cloud data, which is crucial for identifying and segmenting objects with non-uniform distribution characteristics such as roads and vehicles.

[0023] Step 5: The present invention proposes a radial window attention module to aggregate local features and global context information at different levels. Its overall network structure design is highly modular, such as Figure 3 As shown, it can be flexibly inserted into the existing point cloud processing network.

[0024] In this module, in order to better extract the features of distant sparse points, RWA uses a radial window mechanism to divide the local area of ​​the point cloud. Unlike the cubic window partitioning mechanism, it determines the size of the local neighborhood based on the radius from the center point. This radial window partitioning mechanism is the core part of RWA. In this way, RWA can more effectively process point cloud data with uneven density and improve the adaptability and robustness of feature extraction.

[0025] At the same time, spherical convolution is also one of the cores of RWA. RWA can more accurately extract the local geometric features of the point cloud through spherical convolution operations while maintaining the integrity of spatial information. It can directly operate on point cloud data without relying on any grid structure. This convolution method is particularly suitable for processing point cloud data with irregular distribution characteristics. In the road point cloud segmentation task, it can help the model better understand the overall shape and structure of targets such as buildings, vehicles, and pedestrians, thereby improving the accuracy of segmentation.

[0026] In RWA, we use a multi-head attention mechanism to learn global and local features. The multi-head attention mechanism is one of the important components of the Transformer model. It can effectively process sequence data and capture long-distance dependencies. By dividing the input sequence into multiple "heads", each head learns a different part of the input representation, and then merges these representations to capture different aspects of the information. In the complex road point cloud segmentation scenario, the multi-head attention mechanism significantly improves the model performance by extracting multi-scale features and improving the model's ability to capture features of different scales and positions.

[0027] The specific implementation of the multi-head attention mechanism is as follows: (1.2) in, represents the input features of a window, is the linear projection weight is the projection feature.

[0028] Then project the features Divided into h attention heads (i.e. ), and transform it into For each attention head, dot product and weighted sum are performed: (1.3) (1.4) in, represents the features of the kth attention head, and is the corresponding attention weight. Finally, the features of all attention heads are concatenated and the final linear projection and weight are applied To get the feature output : (1.5) (1.6) In addition, in order to reduce the amount of computation of the core computing part of the Transformer in the RWA module, we propose to use the jump self-attention mechanism to calculate the Transformer self-attention part, which improves the computational efficiency of the Transformer model by reducing redundant calculations in attention calculations. Specifically, each layer of the Transformer consists of a multi-head attention (MSA) module and a multi-layer perceptron (MLP) module.

[0029] The study found that there is a high correlation between the output representation ZMSA in the MSA blocks of each layer. Based on this research result, a method is proposed to reuse the ZMSA from the previous layer as the input of the SKIPAT parameter function and skip the MSA operation in one or more subsequent layers of the Transformer. The features of the SKIPAT parameter function output are then used as the input of the MLP module. This mechanism allows the model to "jump" between different layers, that is, lower layers can interact directly with higher layers without the need to pass information layer by layer.

[0030] The specific implementation of this "skip" mechanism is as follows: In order to ensure that cross-layer reuse of self-attention blocks does not degrade performance, a simple parameter function is introduced, which runs faster and has improved performance. The SKIPAT parameter function consists of two linear layers (Fully Connected, FC) and a depthwise convolution (DwC). The point cloud block embedding vector is input into the first linear layer FC1 to expand the channel dimension: .

[0031] Connect DwC later Convolution kernel is used to extract the relationship features between point cloud elements: , before performing the deep convolution operation, the matrix needs to be reshaped into a feature tensor.

[0032] Next, the output of DwC is converted back into a matrix and input into the last fully connected layer FC2: , which reduces the channel dimension to the initial dimension ,Finally, an efficient channel attention module (ECA) is used after FC2 to enhance the cross-channel dependencies and obtain the output The formula is as follows: (1.7) In addition, this mechanism enables the model to process information of different scales more flexibly. When processing road point cloud segmentation, objects of different sizes may require different levels of detail processing, and the jumping self-attention mechanism can better adapt to this demand. For example, large objects (such as vehicles) and small objects (such as traffic signs) can be effectively identified and segmented at the same time without adding too much computational burden, improving the efficiency of road point cloud segmentation and achieving faster road point cloud segmentation.

[0033] The radial window attention module designed in this paper brings new possibilities to point cloud processing. It not only improves processing efficiency, but also enhances the model's ability to understand and analyze complex point cloud data through precise feature extraction and dependency capture. With further research and application, RWA is expected to play a greater role in multiple fields such as autonomous driving, robot perception, and virtual reality.

[0034] In this invention, the training result with the highest mIoU value in the validation set is selected to perform semantic segmentation on the test set. The segmentation results based on the Semantickitti dataset are as follows: Figure 4 As shown in Figure 2, the segmentation results based on the Nuscenes dataset are as follows: Figure 5 shown.

[0035] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0036] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A road scene semantic segmentation method based on lidar point cloud data, characterized in that: The following steps are involved: S1, preprocessing of laser radar raw point cloud, using sparse tensor to represent point cloud data; S2. Build a Transformer-based lightweight road point cloud semantic segmentation network RWAFormer; S3: Pre-training and parameter optimization of the Transformer-based lightweight road point cloud semantic segmentation network RWAFormer built in step S2; S4: Input the preprocessed test set data set into the trained road point cloud semantic segmentation method network in S3, and output the final semantic category.

2. The road scene semantic segmentation method based on laser radar point cloud data according to claim 1, characterized in that: In step S1, the method for preprocessing the original point cloud of the laser radar is as follows: the input point cloud data is first encoded by a sparse tensor, and each point is encoded into a high-dimensional vector to retain the sparsity of the point cloud.

3. The method for semantic segmentation of road scenes based on laser radar point cloud data as claimed in claim 2, characterized in that: The point cloud is represented by a sparse tensor divided into two parts: a coordinate matrix and a feature matrix.

4. The method for semantic segmentation of road scenes based on laser radar point cloud data according to claim 1, characterized in that: In step S2, in RWAFormer, a sparse tensor feature encoding module is introduced. Through its unique continuous convolution operation and sparse tensor encoding mechanism, feature extraction is achieved while retaining the intrinsic characteristics of the point cloud. Then, a radial window attention module is designed. Through the radial window attention mechanism, the information of the long-distance sparse point cloud is effectively aggregated to the adjacent dense area.

5. The method for semantic segmentation of road scenes based on laser radar point cloud data as claimed in claim 4, characterized in that: In step S2, when the sparse tensor feature encoding module extracts features, the point cloud data after sparse tensor feature encoding is subjected to feature extraction through 8 consecutive Minkowski convolution layers. Between these convolution layers, activation functions and regularization layers are interspersed to enhance the nonlinear fitting ability of the network and prevent overfitting.

6. The method for semantic segmentation of road scenes based on laser radar point cloud data as claimed in claim 4, characterized in that: In step S2, the steps of RWA aggregation features are as follows: S21, firstly, the input local features are used to dynamically select the size of the local neighborhood according to the radius of the center point using a radial window mechanism; S22. Then, the input features are subjected to spherical convolution. By designing a convolution kernel adapted to the sphere, the spherical convolution can more accurately extract the local geometric features of the point cloud while maintaining the integrity of the spatial information. This convolution method is particularly suitable for processing point cloud data with irregular distribution characteristics. S23. After the spherical convolution, RWA applies a multi-head attention mechanism, which enables the model to learn information in multiple different feature subspaces simultaneously, which helps to capture the complex dependencies between different features.

7. The method for semantic segmentation of road scenes based on laser radar point cloud data according to claim 1, characterized in that: In step S2, the vector attention mechanism is combined with the U-Net encoder-decoder framework, where the U-net structure contains 5 encoders and 5 decoders. The encoder consists of a downsampling module, STFE and the RWA proposed in this paper to capture features, and the decoder consists of an upsampling module, STFE and RWA to map features.

Images