A three-dimensional point cloud semantic segmentation method using space-time information

Abstract

The application discloses a three-dimensional point cloud semantic segmentation method using space-time information. Firstly, a three-dimensional feature extraction backbone network is used to obtain a feature map at a single time point, then the feature map is strengthened in a space domain through a local consistency loss function, and finally, the time sequence feature map is fused in a time domain through a long short-term memory mechanism, so that an enhanced three-dimensional point cloud semantic segmentation output result is obtained. The application can be applied to various types of existing backbone networks, and the performance of the backbone networks can be further enhanced by introducing only a small amount of additional network parameters and calculation amount, without the need of retraining the entire backbone network, and the application has the characteristics of lightness, universality and high efficiency.

Description

Technical Field

[0001] This invention relates to a three-dimensional point cloud semantic segmentation method in the field of autonomous driving, and more particularly to a three-dimensional point cloud semantic segmentation method utilizing spatiotemporal information. Background Technology

[0002] 3D point cloud semantic segmentation refers to assigning a semantic category label, such as vehicle, person, or traffic sign, to each point in the input 3D point cloud data; it falls under the category of classification tasks. In recent years, deep learning has developed rapidly, learning effective features from large amounts of training data, which has greatly promoted the development of classification tasks. Current research in the field of 3D point cloud semantic segmentation focuses on two aspects: firstly, mining structural features between points within a single frame of input point cloud; secondly, utilizing temporal information across multiple frames of input point cloud; and thirdly, improving the loss function for network training.

[0003] In single-frame point cloud processing, representative methods include PointNet++, KPConv, Cylinder3D, and SphereFormer. The main differences between these methods lie in their point cloud representation methods and the design of their corresponding core operators, such as 3D point representation, 3D voxel representation, graph representation, 2D image representation, and hybrid representation. The design of the core operators in the encoder is based on the point cloud representation method; correspondingly, representative methods include 3D point convolution, 3D voxel convolution, graph convolution, and 2D mesh convolution.

[0004] During autonomous driving, data collection often involves acquiring a series of time-divided data frames as the vehicle moves. While the sparsity of LiDAR data within each frame is relatively high, the temporal frames exhibit continuity. Therefore, utilizing temporal information for point cloud semantic segmentation is a significant research direction. Mainstream methods fall into two categories: multi-frame input methods and recurrent neural network (RNN)-based methods. The former involves a single input of multiple point cloud frames, processing the input point cloud temporally using convolution, attention, and other techniques; representative methods include SpSequenceNet, PSTNet, and MinkowskiNet. The latter involves a single input of a single point cloud frame, using RNNs, GRUs, and other mechanisms to transmit information temporally; representative methods include LatticeNet. Because current temporal methods often employ relatively basic modules (such as ordinary 2D convolution and max pooling) to construct the overall network, they are not superior to single-frame methods in terms of accuracy and speed.

[0005] On the other hand, in the loss function part of network training, semantic segmentation is a classification problem, and the most basic loss function is the cross-entropy loss function in multi-class classification problems. Based on this, methods such as WCE and OHEM reallocate the weights in the loss function according to the imbalance of the number of samples in each class in the training data; Lovasz constructs the network's loss function using the mIoU metric of semantic segmentation; methods such as RFCC and SceneEncoder supervise the network's output at multiple scales; and the NCM method statistically analyzes the adjacency probabilities of classes within local regions of the scene to construct the loss function.

[0006] Among the methods introduced above, the single-frame-based method lacks the utilization of temporal information, and the temporal information-based method has shortcomings in spatiotemporal feature modeling. Summary of the Invention

[0007] To address the problems and needs in the background technology, this invention proposes a three-dimensional point cloud semantic segmentation method based on temporal feature fusion and spatial local consistency prior. This method can be directly applied to various types of single-frame backbone networks, and its performance can be improved by introducing only a small number of additional network parameters and computational costs. It does not require retraining the entire network and has the characteristics of being lightweight, versatile, and efficient.

[0008] The purpose of this invention is to enhance the performance of 3D point cloud semantic segmentation networks based on temporal feature fusion and spatial local consistency, thereby providing high-precision LiDAR point cloud semantic segmentation results for autonomous driving. For a single-frame 3D backbone network, the feature map is strengthened by training with a spatial local consistency loss function, and then the feature maps at different times are fused in the temporal domain through a long short-term memory mechanism to finally obtain the enhanced 3D point cloud semantic segmentation output.

[0009] The objective of this invention is achieved through the following technical solution:

[0010] Step 1: Use the encoder and decoder in the 3D point cloud semantic segmentation network as the backbone network for 3D feature extraction. Input the temporal continuous frame point cloud data into the 3D feature extraction backbone network and output the temporal feature map set.

[0011] Step 2: Train the spatiotemporal enhancement network module using the temporal feature map set to obtain the trained spatiotemporal enhancement network module; connect the 3D feature extraction backbone network with the trained spatiotemporal enhancement network module to form a 3D point cloud semantic segmentation enhancement network;

[0012] Step 3: Input the temporal continuous frame point cloud data to be processed into the 3D point cloud semantic segmentation enhancement network in sequence to obtain the enhanced temporal continuous frame point cloud semantic segmentation output results.

[0013] The input to the 3D point cloud semantic segmentation network is a single-frame image or multiple-frame images.

[0014] The spatiotemporal augmentation network module includes a kernel convolution module and a long short-term memory temporal fusion module. The kernel convolution module takes the spatiotemporally augmented feature map of the previous frame and the initial feature map of the current frame as input. The kernel convolution module outputs the input features and state features of the current frame and uses them as input to the long short-term memory temporal fusion module. The long short-term memory temporal fusion module outputs the spatiotemporally augmented feature map of the current frame.

[0015] In the kernel convolution module, each point in the initial feature map of the current frame is convolved with its neighboring points in the initial feature map of the current frame to obtain the intermediate feature map of the current frame and is recorded as the input feature of the current frame; each point in the initial feature map of the current frame is convolved with its neighboring points in the spatiotemporally enhanced feature map of the previous frame to obtain the enhanced intermediate feature map of the previous frame and is recorded as the state feature of the current frame.

[0016] The Long Short-Term Memory temporal fusion module fuses the current frame input features with the current frame state features to obtain the spatiotemporally enhanced feature map of the current frame.

[0017] In step 2, the network loss function of the spatiotemporal enhancement network module is composed of the semantic segmentation cross-entropy loss and the local class consistency loss.

[0018] The local class consistency loss L loc The formula is as follows:

[0019]

[0020]

[0021]

[0022]

[0023]

[0024] in, This means that a point i located within a category region and correctly classified is forced to output the same category as its neighbor j. This means that the internal region of each category must contain only points of that category, and no points of other categories. It is the output class probability distribution at the i-th point. M is the output class probability distribution of the j-th neighbor of the i-th point; val It is an effective region mask, which contains points located within the category region that are correctly classified. Points located on the edge of the category or misclassified It is the mask value corresponding to the i-th point; M is the number of neighboring points of each point. It is the truth class encoding at the i-th point. It is the truth class encoding of the j-th neighbor of the i-th point, NCM pred and NCM gt These are the predicted and ground truth values ​​of the neighborhood category co-occurrence matrix, respectively, where C is the number of semantic categories. It is the outer product of vectors, |||2 is the 2-norm of the vector, and || is the modulo operation. It is the predicted value of the element in the i-th row and j-th column of the neighborhood category co-occurrence matrix. It is the truth value of the element in the i-th row and j-th column of the neighborhood class co-occurrence matrix.

[0025] By adopting the above technical solution, the beneficial effects of the present invention are:

[0026] This invention simultaneously absorbs the advantages of a single-frame-based 3D backbone network and 3D point cloud temporal information, and further improves the performance of the 3D point cloud semantic segmentation network by utilizing the temporal information of continuous frame point clouds in the time domain.

[0027] This invention utilizes the local class consistency existing within the internal regions of each category in the scene to construct a loss function, thereby improving the output results of the 3D point cloud semantic segmentation network from the perspective of local class consistency in the spatial domain;

[0028] This invention can be directly applied to various types of backbone networks. It can further enhance their performance by introducing only a small number of additional network parameters and computational costs. It does not require retraining the entire backbone network and has the characteristics of being lightweight, versatile, and efficient. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the spatiotemporal enhancement network module and its process of the present invention;

[0030] Figure 2 This is a schematic diagram illustrating the local category consistency prior of the present invention. Detailed Implementation

[0031] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Taking the publicly available autonomous driving dataset SemanticKITTI as an example, sequences 00-07 and 09-10 are selected as the training set, and sequence 08 is selected as the validation set. The LiDAR device used is a 64-line Velodyne 3D LiDAR with a frequency of 10Hz. One NVIDIA 3090 graphics card was used in the experiment, and the batch size was set to 2 during training and 1 during inference.

[0032] The specific implementation steps of this invention are as follows:

[0033] Step 1: Use the encoder and decoder in the 3D point cloud semantic segmentation network as the backbone network for 3D feature extraction to extract the initial feature map. That is, delete the semantic segmentation head in the 3D point cloud semantic segmentation network, input the temporal continuous frame point cloud data into the 3D feature extraction backbone network, and output the temporal feature map set.

[0034] The input to the 3D point cloud semantic segmentation network is a single-frame image or multiple-frame images. In this embodiment, taking Cylinder3D as an example, the input point cloud is first voxelized in a 3D cylindrical coordinate system. Then, the voxelized data is input into a 3D sparse convolutional network spconv to extract the features of each voxel. After that, the index correspondence between each voxel and the original input point is calculated, and the features at each voxel are projected to each point to obtain the feature map of the input point cloud.

[0035] Step 2: Train the spatiotemporal enhancement network module using the temporal feature map set to obtain the trained spatiotemporal enhancement network module; connect the 3D feature extraction backbone network with the trained spatiotemporal enhancement network module to form a 3D point cloud semantic segmentation enhancement network;

[0036] like Figure 1 As shown, the spatiotemporal augmentation network module includes a kernel convolution (KPConv) module and a long short-term memory (LSTM) temporal fusion module. The KPConv module and the LSTM temporal fusion module are connected. The input to the KPConv module is the spatiotemporally augmented feature map of the previous frame (i.e., the previous frame output of the LSTM temporal fusion module) and the initial feature map of the current frame (i.e., the output of the 3D feature extraction backbone network). The KPConv module outputs the current frame input features and the current frame state features, which are then used as input to the LSTM temporal fusion module. The LSTM temporal fusion module outputs the spatiotemporally augmented feature map of the current frame and stores it in memory as input for the next frame, avoiding repeated inference in the next time step. For the first frame of the sequence, the initial feature map of the current frame is reused as the spatiotemporally augmented feature map of the previous frame.

[0037] In the kernel convolution module, each point in the initial feature map of the current frame is convolved with its neighboring points in the same initial feature map to obtain the intermediate feature map of the current frame, which is recorded as the input feature of the current frame. Similarly, each point in the initial feature map of the current frame is convolved with its neighboring points in the spatiotemporally enhanced feature map of the previous frame to obtain the enhanced intermediate feature map of the previous frame, which is recorded as the state feature of the current frame. Neighboring points are obtained through a 3D k-nearest neighbor query. In this embodiment, the number of network parameters in the 3D backbone network Cylinder3D is 223.6M, while the number of parameters in the spatiotemporally enhanced network module proposed in this invention is only 1.5M.

[0038] The Long Short-Term Memory temporal fusion module fuses the input features of the current frame with the state features of the current frame to obtain the spatiotemporally enhanced feature map of the current frame.

[0039] In step 2, the network loss function of the spatiotemporal augmentation network module is composed of the semantic segmentation cross-entropy loss and the local class consistency loss.

[0040] The overall loss function L is as follows, where λ is a hyperparameter, and in this embodiment, it is taken as 0.3:

[0041] L = L ce +λL loc

[0042] Local class consistency loss L loc The formula is as follows, and the diagram illustrating the intra-class local consistency prior on which this loss is based is shown below. Figure 2 (a) and Figure 2 As shown in (b):

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] in, This means that a point i located within a category region and correctly classified is forced to output the same category as its neighbor j. This means that the internal region of each category must contain only points of that category, and no points of other categories. It is the output class probability distribution at the t-th point. M is the output class probability distribution of the j-th neighbor of the i-th point; val It is an effective region mask, which contains points located within the category region that are correctly classified. Points located on the edge of the category or misclassified It is the mask value corresponding to the i-th point; M is the number of neighboring points of each point. The truth class encoding at the i-th point. It is the truth class encoding of the j-th neighbor of the i-th point, NCM pred and NCM gt These are the predicted and ground truth values ​​of the neighborhood category co-occurrence matrix, respectively, where C is the number of semantic categories. It is the outer product of vectors, |||2 is the 2-norm of the vector, representing the square root of the sum of the squares of all elements of the vector, and || is the modulo operation, representing the number of elements in the set. It is the predicted value of the element in the i-th row and j-th column of the neighborhood category co-occurrence matrix. It is the truth value of the element in the i-th row and j-th column of the neighborhood class co-occurrence matrix.

[0049] Semantic segmentation cross-entropy loss function L ce as follows:

[0050]

[0051] in, Let y represent the predicted probability that the i-th point belongs to the c-th class. ic w represents the true probability that the i-th point belongs to the c-th class. c This represents the weight of class c, which is the reciprocal of the percentage of total points of that class in the entire training set. C is the total number of classes, and N is the number of points in the current frame.

[0052] Step 3: Input the temporal continuous frame point cloud data to be processed into the 3D point cloud semantic segmentation enhancement network in sequence to obtain the enhanced temporal continuous frame point cloud semantic segmentation output results.

[0053] In practice, the network is trained by training two parts sequentially: a 3D backbone network and a spatiotemporal augmentation network. For the 3D backbone network, it is trained to convergence using its original configuration. Alternatively, pre-trained network parameters can be directly imported and then fixed. For the spatiotemporal augmentation network, keyframes are randomly sampled from the temporal training set. Each keyframe and its preceding N frames are used as a training window. Within each window, starting from frame t, the output at time t is obtained through steps 1 to 3. The loss function is calculated at time t, and the output at time t is used as the input at time t+1, processing this process sequentially up to the last frame within the window. In this stage, the batch size is set to 2, the AdamW optimizer is used, the initial learning rate is set to 0.0001, cosine annealing is used to control the learning rate, the maximum learning rate is set to 0.001, and the training lasts for 20 epochs.

[0054] Table 1 shows the comparison of the classification performance of the present invention and the Cylinder3D backbone network on the SemanticKITTI validation set. The indicators in Table 1 are the Intersection over Union (IoU) and the average IoU for each category.

[0055] Table 1 compares the classification performance of the present invention with that of the Cylinder3D backbone network.

[0056]

[0057] Table 2 shows the comparison of model size and inference time between the present invention and the Cylinder3D backbone network on the SemanticKITTI validation set. TTA (Test-Time Augmentation) is a commonly used result augmentation method when testing network models. TTA-5 means that each frame of input data is subjected to 5 random data augmentation transformations before inference, and the average of the 5 inference results is taken as the output result.

[0058] Table 2 compares the model size and inference time of the present invention with those of the Cylinder3D backbone network.

[0059] Cylinder3D 223.6M 65.3 231ms +TTA-5 223.6M 66.1 954ms +Invention 225.1M 67.7 299ms

[0060] Table 3 shows the classification performance of this invention compared with various backbone networks on the SemanticKITTI validation set. The index in Table 3 is the average intersection-union ratio (mIoU) for each category.

[0061] Table 3 compares the classification performance of this invention with that of various backbone networks.

[0062] PVKD 68.8 70.8 2.0 2DPASS 70.8 72.0 1.2 KPRNet 62.9 65.7 2.8 SpSequenceNet 43.2 45.4 2.2

[0063] The above results show that the present invention can improve backbone network performance by introducing only a small number of additional network parameters and time consumption, and is applicable to a variety of backbone networks. It is lightweight, versatile and efficient.

[0064] The above are merely specific embodiments of the present invention and should not be construed as limiting the scope of the present invention. Equivalent variations made by those skilled in the art based on this invention, as well as changes well-known to those skilled in the art, should still fall within the scope of the present invention.

Claims

1. A three-dimensional point cloud semantic segmentation method utilizing spatiotemporal information, characterized in that, It includes the following steps: Step 1: Use the encoder and decoder in the 3D point cloud semantic segmentation network as the backbone network for 3D feature extraction. Input the temporal continuous frame point cloud data into the 3D feature extraction backbone network and output the temporal feature map set. Step 2: Train the spatiotemporal augmentation network module using the temporal feature map set to obtain the trained spatiotemporal augmentation network module; A 3D point cloud semantic segmentation enhancement network is formed by connecting a 3D feature extraction backbone network with a trained spatiotemporal enhancement network module. The spatiotemporal augmentation network module includes a kernel convolution module and a long short-term memory temporal fusion module. The kernel convolution module takes the spatiotemporally augmented feature map of the previous frame and the initial feature map of the current frame as input. The kernel convolution module outputs the input features and state features of the current frame and uses them as input to the long short-term memory temporal fusion module. The long short-term memory temporal fusion module outputs the spatiotemporally augmented feature map of the current frame. In the kernel convolution module, each point in the initial feature map of the current frame is convolved with its neighboring points in the initial feature map of the current frame to obtain the intermediate feature map of the current frame and is recorded as the input feature of the current frame; each point in the initial feature map of the current frame is convolved with its neighboring points in the spatiotemporally enhanced feature map of the previous frame to obtain the enhanced intermediate feature map of the previous frame and is recorded as the state feature of the current frame. In step 2, the network loss function of the spatiotemporal enhancement network module is composed of the sum of semantic segmentation cross-entropy loss and local class consistency loss; The aforementioned local class consistency loss The formula is as follows: in, This means that a point i located within a category region and correctly classified is forced to output the same category as its neighbor j. This means that the internal region of each category must contain only points of that category, and no points of other categories. It is the output class probability distribution at the i-th point. It is the output class probability distribution of the j-th neighbor of the i-th point; It is an effective region mask, which contains points located within the category region that are correctly classified. Points located on the edge of the category or misclassified ; It is the mask value corresponding to the i-th point; It is the number of neighboring points of each point. It is the truth class encoding at the i-th point. It is the truth class encoding of the j-th neighbor of the i-th point. and These are the predicted and true values ​​of the neighborhood class co-occurrence matrix, respectively. It is the number of semantic categories. It is the outer product of vectors. It is the 2-norm of the vector. It is a modulo operation. It is the predicted value of the element in the i-th row and j-th column of the neighborhood category co-occurrence matrix. It is the truth value of the element in the i-th row and j-th column of the neighborhood category co-occurrence matrix; Step 3: Input the temporal continuous frame point cloud data to be processed into the 3D point cloud semantic segmentation enhancement network in sequence to obtain the enhanced temporal continuous frame point cloud semantic segmentation output results.

2. The three-dimensional point cloud semantic segmentation method utilizing spatiotemporal information according to claim 1, characterized in that, The input to the 3D point cloud semantic segmentation network is a single-frame image or multiple-frame images.

3. The three-dimensional point cloud semantic segmentation method utilizing spatiotemporal information according to claim 1, characterized in that, The Long Short-Term Memory temporal fusion module fuses the current frame input features with the current frame state features to obtain the spatiotemporally enhanced feature map of the current frame.

Images