A point cloud semantic segmentation method and apparatus

Abstract

This invention discloses a point cloud semantic segmentation method and apparatus. It includes: collecting point cloud data from a scene as a dataset; constructing a semantic segmentation network structure that integrates temporal information by using 3D sparse convolution as an encoder / decoder and designing an inter-frame local attention module based on Point Transformer; training the semantic segmentation network based on the acquired dataset to obtain a model for semantic segmentation; and performing semantic segmentation on the point cloud data to be segmented using the semantic segmentation model. This invention can handle large-scale outdoor scene tasks, and the addition of an inter-frame local attention module can solve the problems of low segmentation accuracy for occluded objects and a few categories, while avoiding interference from useless information caused by the introduction of global features in general Transformers.

Description

Technical Field

[0001] This invention relates to a semantic segmentation method and apparatus for LiDAR in autonomous driving scenarios, specifically to a point cloud semantic segmentation method and apparatus based on Point Transformer fusion of temporal information. Background Technology

[0002] Due to the sparsity of point clouds, point cloud semantic segmentation methods based on single frames do not perform well for a few categories. At present, the application of temporal information in point cloud semantic segmentation is not very common. Some methods use time as a dimension and then use sparse 4D convolution to process the 4D occupied grid, but the memory consumption and computational cost are huge. Alternatively, methods such as RNN and LSTM also have problems such as parallel operation and network convergence.

[0003] In recent years, the Transformer algorithm, originally applied to natural language processing, has begun to be used in 3D scene understanding. However, due to the high computational cost of the Transformer and the fact that its self-attention mechanism focuses on global information, it introduces too much useless information when applied to point cloud semantic segmentation, resulting in unsatisfactory segmentation results. This invention combines a deep learning backbone network with the Point Transformer and improves the attention mechanism by utilizing the semantic information of the previous frame, making the segmentation results more accurate. Summary of the Invention

[0004] To address the problems existing in the background technology, the present invention provides a point cloud semantic segmentation method and apparatus, which is suitable for low-speed motion scenarios and can effectively overcome the data sparsity problem of single-frame point clouds, achieving higher accuracy than single-frame point cloud segmentation.

[0005] The technical solution adopted in this invention is as follows:

[0006] I. A point cloud semantic segmentation method

[0007] Includes the following steps:

[0008] Step 1) Collect point cloud data from the scene using data acquisition equipment as a dataset;

[0009] Step 2) Using 3D sparse convolution as the encoder and decoder, design an inter-frame local attention module based on Point Transformer to construct a semantic segmentation network structure that integrates temporal information;

[0010] The point cloud semantic segmentation network that integrates temporal information includes a data preprocessing module, a feature encoding module, an inter-frame local attention module, a feature encoding / decoding module, and a post-processing module connected in sequence.

[0011] Step 3) Train the semantic segmentation network constructed in step 2) based on the dataset obtained in step 1) to obtain a model for semantic segmentation;

[0012] Step 4) Use a semantic segmentation model to perform semantic segmentation on the point cloud data to be segmented.

[0013] Step 1) specifically involves first determining the semantic segmentation categories based on task requirements (taking autonomous driving scenarios as an example, the main categories of interest include vehicles, people, drivable roads, sidewalks, and other obstacles); then acquiring multiple sequences by using LiDAR and cameras to collect data from different areas of the same scene, with each sequence including point cloud data collected by LiDAR and images collected by the camera; finally, constructing the correspondence between point clouds and image pixels based on camera intrinsic and extrinsic parameters, labeling the point cloud data by category, and obtaining the ground truth label corresponding to each point, thereby completing the dataset collection.

[0014] The LiDAR and camera must maintain the same frequency so that both devices can capture the scene at the same moment.

[0015] In step 2), the data preprocessing module includes preprocessing operations, an MLP layer, and a cylindrical voxelization layer. The preprocessing operation is to convert the Cartesian coordinates of the input point cloud into cylindrical coordinates, and then calculate the voxel index of each point and the initial features of each point through manually set hyperparameters.

[0016] The input data preprocessing module converts the current frame point cloud and the previous frame point cloud in the sequence into voxels and outputs the features and indexes of each non-empty voxel in the current frame and the previous frame.

[0017] The feature encoding module and feature decoding module in step 2) are designed based on the Encoder-Decoder network structure in the UNet network, and an inter-frame local attention module is set between the feature encoding module and the feature decoding module.

[0018] The output of the data preprocessing module is input into the feature encoding module to obtain the feature vector F of the current frame. t and the feature vector F of the previous frame t-1 The feature vector F of the current frame t and the feature vector F of the previous frame t-1 The input inter-frame local attention module outputs a feature vector F. t Then, the signal is input into the post-processing module via the feature decoding module.

[0019] The inter-frame local attention module in step 2) consists of two linear layers and one inter-frame local attention layer. The inter-frame local attention layer includes three linear learning layers, two neighborhood search layers, a position encoding layer, and a learnable weight layer.

[0020] The inter-frame local attention module uses the index positions of the current frame and the previous frame to find the k-neighborhood of each non-empty voxel in the current frame corresponding to the previous frame, and then updates the features of the current frame through the weight matrix. Specifically, it includes the following steps:

[0021] 1) Transfer the feature vector F of the current frame t and the feature vector F of the previous frame t-1 After inputting the first linear layer, the feature vector F of the current frame is obtained respectively. t ′ and the feature vector F of the previous frame t-1 ′;

[0022] 2) Linear learning layer: Feature vector F of the current frame t The Q vector is obtained through a linear transformation of the first linear learning layer; the feature vector F of the previous frame. t-1 The K vector and V vector are obtained by linear transformations through the second and third linear learning layers, respectively.

[0023] Where Q∈R n2×C , K∈R n1×C , V∈R n1×C n1 and n2 represent the number of non-empty voxels in the previous frame and the current frame, respectively, and C represents the number of feature channels;

[0024] 3) Neighborhood search layer:

[0025] The K vector is used to perform a neighborhood search in the first neighborhood search layer to obtain the k neighborhoods of the previous frame corresponding to each effective voxel in the current frame, and outputs the feature vector K′ and the relative position pr.

[0026] The V vector is processed by the neighborhood search operation of the second neighborhood search layer to obtain the k neighborhoods of the previous frame corresponding to each valid voxel in the current frame, and the feature vector V′ is output.

[0027] Where K′∈R n2×k×C ;V′∈R n2×k×C pr represents the relative positions of k neighborhoods to the center point, where pr∈R n2×k×3 ;

[0028] 4) Position Encoding Layer: The relative position pr output by the first neighborhood search layer is input to the position encoding layer. The position encoding layer unifies the feature dimensions of pr to obtain p. r ′, p r ′∈R n2×k×C

[0029] 5) Learnable weight layer: The learnable weight layer consists of an MLP layer and a softmax function. The MLP layer includes two linear layers and one ReLU nonlinear layer.

[0030] The inputs to the learnable weight layer are vectors Q, K′, and p. r The correlation between the neighborhood and the center point is learned through the MLP layer, and then the weight matrix is ​​obtained through the softmax function.

[0031] 6) Feature Update Layer: The output p of the position encoding layer is updated. r The updated effective voxel features F are obtained by adding the feature vector V' to the feature vector V' and then multiplying it with the weight matrix output by the learnable weight layer. t "

[0032] 7) Effective voxel specific F t "The input is the second linear layer, and the output of the second linear layer is the output of the inter-frame local attention module."

[0033] The post-processing module in step 2) includes a DDCM module, a disintegration and elementization module, and a point-by-point refinement module connected in sequence.

[0034] The output of the feature decoding module is input into the DDCM module, then decomposed by the decomposition and pixelation module. The output of the decomposition and pixelation module is added to the output of the MLP layer in the data preprocessing module and then input into the point-by-point refinement module (MLP layer) to learn features point by point, thus obtaining fine-grained segmentation results.

[0035] The point-by-point refinement module uses each point as a basic unit to obtain fine-grained segmentation results.

[0036] II. A point cloud semantic segmentation device

[0037] Including 3D sensing devices and computing equipment;

[0038] The 3D sensing device includes a lidar sensor and a camera. The lidar sensor is used to acquire point cloud data of the scene. The lidar sensor is not limited and can be either a traditional mechanical type or a lidar of various types, such as solid-state or semi-solid-state lidar. The camera is used to acquire scene images.

[0039] The computing device includes a memory and a processor. The memory stores a computer program that runs on the processor. When the processor executes the computer program, it implements the point cloud semantic segmentation method described above.

[0040] The beneficial effects of this invention are:

[0041] This invention is based on 3D sparse convolution and can handle large-scale outdoor scene tasks. It adds an inter-frame local attention module, which can solve the problems of occluded objects and low segmentation accuracy for a few categories, and avoids the interference of useless information caused by the introduction of global features in general Transformers. Attached Figure Description

[0042] Figure 1 This is a flowchart of the present invention.

[0043] Figure 2 This is a flowchart of a point cloud semantic segmentation network.

[0044] Figure 3 This is a schematic diagram of an asymmetric convolution kernel.

[0045] Figure 4 This is a schematic diagram of the inter-frame local attention layer structure.

[0046] Figure 5 This is a diagram of the semantic segmentation network framework for point clouds.

[0047] Figure 6 This is a schematic diagram of the encoder-decoder structure in the voxel-level branch. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] like Figure 1 As shown in the flowchart, the following is an embodiment of the complete method implemented according to the disclosed content:

[0050] Step 101: Collect point cloud data from the scene as a dataset.

[0051] The data acquisition equipment mainly consists of LiDAR, cameras, and a robot chassis. First, it is necessary to determine the category used for semantic segmentation. Then, the Robot Operating System (ROS) can be used to collect data from different areas on campus, or other methods can be used to obtain the data, as long as the frequency between the devices is consistent and the scene is collected at the same time.

[0052] After data acquisition is complete, the camera's intrinsic and extrinsic parameters need to be calibrated. The camera's intrinsic parameters can be obtained from the factory settings or using common intrinsic parameter calibration methods such as the Zhang Youzheng calibration method. The extrinsic parameters can be obtained using Autoware calibration tools, the PnP algorithm, or other extrinsic parameter calibration methods. After obtaining the intrinsic and extrinsic parameters, the ground truth labels of the point cloud data can be obtained using image-point cloud joint annotation methods. The training set, validation set, and test set are then organized, thus completing the preparation of the dataset.

[0053] Step 102: Using 3D sparse convolution as the encoder and decoder, design an inter-frame local attention module based on Point Transformer to construct a semantic segmentation network structure that integrates temporal information.

[0054] The aforementioned 3D sparse convolutional encoder-decoder uses Cylinder3D as the backbone network, such as... Figure 2As shown, the constructed point cloud semantic segmentation network mainly includes the following parts: The network structure for point cloud semantic segmentation includes a data preprocessing module, a feature encoding and decoding module, an inter-frame local attention module, and a point-by-point refinement module.

[0055] 201) Voxelizing multi-frame point clouds in cylindrical coordinates better aligns with the scanning characteristics of LiDAR. The input consists of point clouds from the previous and current frames. After representing the coordinates as cylindrical coordinates, some hyperparameters are manually set to calculate the voxel index of each point. These hyperparameters include the number of voxels R*A*H, voxelization range thresholds Rmin, Rmax, Amin, Amax, Hmin, and Hmax. R, A, and H represent the number of voxels in the radial, lateral, and z-axis directions, respectively. Rmin, Rmax, Amin, Amax, Hmin, and Hmax represent the minimum and maximum attention distances in the radial direction, the minimum and maximum attention angles in the angular direction, and the minimum and maximum attention heights in the height direction, respectively. After obtaining the voxel index of each point, MLP and max pooling operations are used to obtain the initial features and index of each non-empty voxel.

[0056] 202) Input the initial feature maps of multiple frames into the feature encoding module to obtain high-dimensional feature maps.

[0057] 203) The high-dimensional feature map is input into the inter-frame local attention module to obtain the fused feature map. The inter-frame local attention module consists of two linear layers and one inter-frame local attention layer. The steps of the inter-frame local attention layer are as follows: Figure 4 As shown, the specific steps are as follows:

[0058] Linear learning layer: The input is the feature vector Ft of the current frame and the feature vector F of the previous frame. t-1 Ft is transformed linearly to obtain the Q vector, and the feature vector F of the previous frame is... t-1 After a linear transformation, we obtain vectors K and V, where Q∈R. n2×C , K∈R n1×C , V∈R n1 ×C n1 and n2 represent the number of non-empty voxels in the previous frame and the current frame, respectively, and C represents the number of feature channels.

[0059] Neighborhood search: The vector K is obtained using the sphere neighborhood search method in PointNet++ to obtain the k-neighborhood K′ of the previous frame for each valid voxel in the current frame and the relative position pr of the neighboring points to the center point, K′∈R. n2×k×C The vector V, after undergoing a spherical neighborhood search operation, yields V′, where V′∈R. n2×k×C .

[0060] Location encoding layer: The input is the relative position pr between the neighborhood point and the center point. After the location encoding layer unifies the feature dimensions, p is obtained. r ′, p r ′∈R n2×k×C The positional coding layer is composed of MLP, and its main purpose is to unify the number of feature channels.

[0061] Learnable weight layer: This module has three input components: Q, K′, and p. r ′ is composed of an MLP with two linear layers and a ReLU nonlinearity and a softmax function. The MLP learns the correlation between the neighborhood and the center point, and then the softmax function obtains the weight matrix.

[0062] Feature Update Layer: The input is the weight matrix and the neighborhood features V′. The updated effective voxel features F are obtained by vector multiplication of the weight matrix and the neighborhood features V′. t ′.

[0063] 204) The fused feature map is input into the feature decoding layer and shallow information is obtained again through skip connections.

[0064] Feature encoding and decoding layers based on asymmetric convolutional kernels, such as... Figure 6 As shown. The feature encoding and decoding layers mainly utilize the Encoder-Decoder network structure in the existing UNet network, and adopt the asymmetric convolutional kernels designed in the Cylinder3D method to replace the traditional convolution. By adding the aforementioned asymmetric convolutional blocks, corresponding asymmetric downsampling and upsampling blocks are designed. The asymmetric convolutional blocks use two 3×1×3 and 1×3×3 convolutional kernels instead of the traditional 3×3 convolutional kernels, as shown. Figure 3 As shown, the asymmetric downsampling block first uses an asymmetric convolutional kernel to upsample the input features, learning their high-dimensional features. Then, it performs a 2x downsampling through 3D convolution. Similarly, the asymmetric upsampling block first performs a 2x upsampling through 3D convolution, halving the number of feature channels. It then fuses the upsampled features with the corresponding upsampled features via skip connections before using an asymmetric convolutional kernel for further learning. Furthermore, the output of the feature decoding module is input to the DDCM (Dimension-Decomposition based Context Modeling) module. The DDCM module uses matrix factorization to weight the features using low-rank vectors in three different directions. This module aggregates the features in each direction using small convolutional kernels of 1×1×3, 1×3×1, and 3×1×1 sizes. The weights in each direction are then obtained using the sigmoid function. The accumulated weights are used to integrate the original features, and the output features are used for the final prediction.

[0065] 205) Shallow information is processed through a classification layer and devoxization to obtain the final voxel-level prediction result. The output of the classification layer is based on voxelization. To obtain the original point cloud form, a devoxization operation is required. The solution adopted in this invention is that points located within the same voxel share a single label.

[0066] 206) After the point-by-point refinement module, the voxel-level output and the point-level output are integrated. Specifically, the output of the MLP in step 201 is added to the voxel-level output and then fed into the MLP of the point-by-point refinement module to learn features point by point.

[0067] Step 103: Train a semantic segmentation network based on the obtained dataset to obtain a model for semantic segmentation, such as... Figure 5 As shown

[0068] Step 104: Use a semantic segmentation model to perform semantic segmentation on the point cloud data to be segmented.

[0069] This invention is applicable to outdoor scenes with low-speed movement and can effectively overcome the data sparsity problem of single-frame point clouds, achieving higher accuracy than single-frame point cloud segmentation.

[0070] The present invention also discloses a semantic segmentation device comprising: a 3D sensing device and a computing device.

[0071] The 3D sensing device refers to a lidar sensor used to acquire point cloud data. The lidar sensor is not limited; it can be a traditional mechanical type, or a lidar of various types, such as solid-state or semi-solid-state lidar. When preparing the dataset, a camera is generally required as an additional sensor for point cloud annotation.

[0072] The computing device includes, but is not limited to, memory, a central processing unit (CPU), a graphics processing unit (GPU), any suitable processor, controller, microcontroller, etc., and computer programs stored in the memory and capable of running on the processor.

[0073] The computer program stored in the memory and executable on the processor is characterized in that, when the processor executes the computer program, it is used to implement each step of the point cloud semantic segmentation method.

[0074] Taking a campus setting as an example, the LiVox Horizon LiDAR and Zed binocular camera were used during data acquisition, and the resulting dataset was named the Horizon dataset. The labeled categories include vehicles, people, drivable roads, sidewalks, and other categories. Table 1 compares the results with and without the inter-frame local attention module, showing that the network with the inter-frame local attention module can compensate for the shortcomings of single-frame network segmentation.

[0075] Table 1 Comparison of Horizon test set results

[0076] Inter-frame local attention module MIoU vehicle people the way sidewalk other Do not use 87.327％ 91.04％ 76.10％ 91.98％ 83.43％ 94.09％ use 87.663％ 90.98％ 77.34％ 92.20％ 83.68％ 94.13％

[0077] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this disclosure.

Claims

1. A point cloud semantic segmentation method, characterized in that, Includes the following steps: Step 1) Collect point cloud data from the scene using data acquisition equipment as a dataset; Step 2) Using 3D sparse convolution as the encoder and decoder, design an inter-frame local attention module based on Point Transformer to construct a semantic segmentation network structure that integrates temporal information; The point cloud semantic segmentation network that integrates temporal information includes a data preprocessing module, a feature encoding module, an inter-frame local attention module, a feature encoding / decoding module, and a post-processing module connected in sequence. Step 3) Train the semantic segmentation network constructed in step 2) based on the dataset obtained in step 1) to obtain a model for semantic segmentation; Step 4) Perform semantic segmentation on the point cloud data to be segmented using a semantic segmentation model. In step 2), the feature encoding module and feature decoding module are designed based on the Encoder-Decoder network structure in the UNet network, and an inter-frame local attention module is set between the feature encoding module and the feature decoding module; the output of the data preprocessing module is input to the feature encoding module to obtain the feature vector F of the current frame. t and the feature vector F of the previous frame t−1 The feature vector F of the current frame t and the feature vector F of the previous frame t−1 The input inter-frame local attention module outputs a feature vector F. t The feature decoding module then inputs the data into the post-processing module. The inter-frame local attention module in step 2) consists of two linear layers and one inter-frame local attention layer. The inter-frame local attention layer includes three linear learning layers, two neighborhood search layers, a position encoding layer, and a learnable weight layer. The inter-frame local attention module uses the index positions of the current frame and the previous frame to find the k-neighborhood of each non-empty voxel in the current frame corresponding to the previous frame, and then updates the features of the current frame through the weight matrix. Specifically, it includes the following steps: 1) Transfer the feature vector F of the current frame t and the feature vector F of the previous frame t−1 After inputting the first linear layer, the feature vector F of the current frame is obtained respectively. t ′ and the feature vector F of the previous frame t−1 ′; 2) Linear learning layer: Feature vector F of the current frame t The Q vector is obtained through a linear transformation of the first linear learning layer; the feature vector F of the previous frame. t−1 The K vector and V vector are obtained by linear transformations through the second and third linear learning layers, respectively. Where Q∈R n2×C , K∈R n1×C V∈R n1×C n1 and n2 represent the number of non-empty voxels in the previous frame and the current frame, respectively, and C represents the number of feature channels; 3) Neighborhood search layer: The K vector is used to perform a neighborhood search in the first neighborhood search layer to obtain the k neighborhoods of the previous frame corresponding to each effective voxel in the current frame, and outputs the feature vector K′ and the relative position pr. The V vector is processed by the neighborhood search operation of the second neighborhood search layer to obtain the k neighborhoods of the previous frame corresponding to each effective voxel in the current frame, and the feature vector V′ is output. Among them, K ′ ∈R n2×k×C V ′ ∈ R n2×k×C pr represents the relative positions of k neighborhoods to the center point, where pr∈R n2×k×3 ; 4) Position Encoding Layer: The relative position pr output by the first neighborhood search layer is input to the position encoding layer. The position encoding layer unifies the feature dimensions of pr to obtain p. r ′, p r ′ ∈ R n2×k×C ; 5) Learnable weight layer: The learnable weight layer consists of an MLP layer and a softmax function. The MLP layer includes two linear layers and one ReLU nonlinear layer. The inputs to the learnable weight layer are vectors Q, K′, and p. r The correlation between the neighborhood and the center point is learned through the MLP layer, and then the weight matrix is ​​obtained through the softmax function. 6) Feature Update Layer: The output p of the position encoding layer is updated. r The updated effective voxel features F are obtained by adding the feature vector V' to the feature vector V' and then multiplying it with the weight matrix output by the learnable weight layer. t ′′; 7) Effective voxel specific F t The second linear layer is input, and the output of the second linear layer is the output of the inter-frame local attention module.

2. The point cloud semantic segmentation method according to claim 1, characterized in that, Step 1) specifically involves first determining the semantic segmentation category based on task requirements; then acquiring multiple sequences by using LiDAR and a camera to collect data from different regions of the same scene, with each sequence including point cloud data collected by LiDAR and images collected by the camera; finally, constructing the correspondence between point cloud and image pixels based on camera intrinsic and extrinsic parameters, labeling the point cloud data by category, and obtaining the ground truth label corresponding to each point, thereby completing the dataset collection.

3. The point cloud semantic segmentation method according to claim 1, characterized in that, In step 2), the data preprocessing module includes preprocessing operations, an MLP layer, and a cylindrical voxelization layer. The preprocessing operations are to convert the Cartesian coordinates of the input point cloud into cylindrical coordinates, and then calculate the voxel index of each point and the initial features of each point through manually set hyperparameters. The input data preprocessing module converts the current frame point cloud and the previous frame point cloud in the sequence into voxels and outputs the features and indexes of each non-empty voxel in the current frame and the previous frame.

4. The point cloud semantic segmentation method according to claim 1, characterized in that, The post-processing module in step 2) includes a DDCM module, a disintegration and elementization module, and a point-by-point refinement module connected in sequence. The output of the feature decoding module is input into the DDCM module, then decomposed by the decomposition and pixelation module. The output of the decomposition and pixelation module is added to the output of the MLP layer in the data preprocessing module and then input into the point-by-point refinement module to learn features point by point, thus obtaining fine-grained segmentation results.

5. A point cloud semantic segmentation device, characterized in that, Including 3D sensing devices and computing equipment; The 3D sensing device includes a lidar sensor and a camera. The lidar sensor is used to acquire point cloud data of the scene, and the camera is used to acquire scene images. The computing device includes a memory and a processor, the memory storing a computer program that runs on the processor, and the processor executing the computer program to implement the steps of the point cloud semantic segmentation method according to any one of claims 1 to 4.

Images