Three-dimensional point cloud semantic segmentation method based on object clustering information

Abstract

The invention relates to the technical field of computer vision, in particular to a three-dimensional point cloud semantic segmentation method based on object clustering information. The method comprises the following steps: 1, acquiring three-dimensional point cloud position information of a vehicle surface, 2, processing point cloud data of a laser radar acquired in the step 1, and 3, clustering historical frame point cloud foreground semantic tags obtained through network inference in the step 2, the method comprises the following steps of 1, obtaining a priori clustering label, 4, sending a point subjected to downsampling in the step 2 and KNN nearest point information of the point into a network for feature extraction, 5, carrying out clustering feature fusion on the priori clustering label obtained in the step 3 and the point feature obtained in the step 4, and 6, predicting the point feature obtained in the step 4 and the clustering feature obtained in the step 5. And step 7, performing voting based on voxels by integrating the historical frame point cloud and the semantic tags of the current frame point cloud obtained by the six modules.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a three-dimensional point cloud semantic segmentation method based on object clustering information. Background Art

[0002] As one of the "four major" research tasks in the field of perception, semantic segmentation has important application prospects in the fields of robot vision and unmanned driving. Depending on the sensors used, semantic segmentation can be divided into image semantic segmentation and point cloud semantic segmentation. Image semantic segmentation is to obtain RGB image information through a camera, and use a 2D neural network to extract image features, and finally obtain the category prediction of each pixel in the image; three-dimensional point cloud semantic segmentation is to use a three-dimensional lidar to obtain point cloud information on the surface of an object, extract point cloud features through a 3D neural network, and finally realize the category prediction of each point in the point cloud.

[0003] In the past, semantic segmentation algorithms based on LiDAR points would have inconsistent predictions for the categories of foreground object points. In addition, most existing LiDAR semantic segmentation algorithms are independent in segmenting laser point clouds at different times, and do not establish temporal connections, resulting in inconsistent semantic segmentation results in terms of timing. Moreover, since images are easily affected by changes in lighting and it is difficult to identify occluded objects or distant objects, LiDAR is not easily affected by the environment, has excellent azimuth resolution, and can accurately obtain geometric information of scenes and objects in the environment. Therefore, the use of LiDAR-based 3D point cloud semantic segmentation is very necessary in the field of unmanned driving.

[0004] Therefore, a three-dimensional point cloud semantic segmentation method based on object clustering information is provided to solve the above problems. The temporal association of the laser point cloud is utilized to obtain the clustering prior information of the foreground object of the current point cloud before the current point cloud is sent to the network, and the classification results and classification consistency of the foreground object are adaptively optimized using this information. In addition, in order to make full use of the temporal information, a voting mechanism is used to vote on the semantic segmentation results of the historical point cloud and the current point cloud, so as to obtain the final prediction result of the current point cloud, in order to cope with the situation where the temporal results of the point cloud at different times are inconsistent, which helps to achieve better performance compared with the existing semantic segmentation methods. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a 3D point cloud semantic segmentation method based on object clustering information to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a 3D point cloud semantic segmentation method based on object clustering information, comprising the following steps:

[0007] Step 1: Collect data on the three-dimensional point cloud position information of the vehicle surface through laser radar;

[0008] Step 2: Process the point cloud data of the laser radar collected in step 1, convert the historical frame point cloud to the current frame coordinate system using position transformation, and downsample the multi-frame point cloud voxelization, and use KNN to obtain the neighboring points of the downsampled points;

[0009] Step 3: The foreground semantic label of the historical frame point cloud obtained by network inference in step 2 is transferred to the current frame and clustered as prior semantic information to obtain a priori clustering label;

[0010] Step 4: Send the downsampled points in step 2 and their KNN nearest point information into the network for feature extraction;

[0011] Step 5: Perform cluster feature fusion on the prior cluster labels obtained in step 3 and the point features obtained in step 4;

[0012] Step 6: Predict the point features obtained in step 4 and the cluster features obtained in step 5;

[0013] Step 7: It is used to integrate the semantic labels of the historical frame point cloud and the current frame point cloud obtained by the above 6 modules, and vote based on voxels.

[0014] Preferably, the three-dimensional point cloud data processing module specifically includes the following steps:

[0015] Step 2.1: Perform semantic segmentation on the multi-frame point cloud, align the coordinate systems of the multi-frame point cloud, and use the laser radar's posture transformation to transform the point cloud coordinates at time tk to time t. The transformation model is: in represents the position transformation of the laser radar from time tk to time t, P t-k represents the 3D coordinates of the point cloud in the coordinate system at time tk, P t Represents the 3D coordinates of the point cloud in the coordinate system at time t;

[0016] Step 2.2: voxelize the point cloud, that is, convert the original point cloud data into standard voxels with a spatial resolution of W×L×H, where W represents the horizontal width, L represents the horizontal length, and H represents the vertical height. Each voxel only retains one 3D point information. The information of each 3D point includes intensity information, x, y, z, and distance. Use KNN to search the nearest N points for the voxelized downsampled points.

[0017] Preferably, the priori clustering label acquisition module specifically includes the following steps:

[0018] Step 3.1: Remap the historical frame semantic labels into two categories: foreground and background. In the current frame coordinate system, first unlabel the semantic labels of the voxels, then use the labels of the historical points to update the voxel labels containing the historical points. The category of each voxel grid is the category of the multiple points contained in the grid. Finally, the voxel labels are assigned to the points of the current frame, thereby obtaining the prior semantic labels of the current frame.

[0019] Step 3.2: Use the DBSCAN clustering algorithm to cluster the foreground objects in the current frame, and cluster the foreground labeled points and unlabeled points at the same time; in addition, after clustering, discard the clusters that do not contain foreground labeled points, thereby filtering out background clustering noise.

[0020] Preferably, the feature encoding and analysis unit in the point cloud data feature extraction module performs feature encoding on the original point cloud, uses a linear layer to encode the 5-dimensional features of the original 3D points, makes a difference between the original features of each point and its N neighboring points, and sends them to a multi-layer perceptron (MLP) for encoding to obtain N such features, then performs maximum pooling on these N features to obtain the local features of the point, and then fuses the local features with the encoded features obtained by the linear layer to obtain the final point features. The encoded feature fusion model is: i =LN([LN(h i ),maxMLP(h j -h i )]), where LN represents the linear layer, h i is the original feature of the current point, h j is the neighboring point p i The original characteristics of

[0021] Preferably, the point cloud data feature extraction module backbone network analysis unit first sends the point feature to the WI module, that is, according to the voxel coordinates of the point, the point feature is projected onto the xy plane, and the point feature contained in each pixel of the xy plane is averaged and pooled to obtain the pixel feature; secondly, the 2D convolutional network is used to perform a convolution operation on the 2D feature map, and the activation function ReLU is selected as the feature activation layer; then the feature of the 2D feature map is returned to each point, and the feature of the feature map pixel is directly assigned to the 3D point projected to the pixel; finally, the point feature is sent to the MLP to enhance the channel dimension. The above process model is: G = F + WI (BN (F)), F = G + MLP (BN (G)), where F∈R F×N is the feature of the point, and G is the feature of the point after passing through the WI module.

[0022] Preferably, the multi-head self-attention mechanism is used in the multi-head self-attention mechanism analysis unit of the adaptive clustering feature fusion module, wherein the attention mechanism is given a query matrix Q, a key matrix K and a value matrix V, and the attention function calculates the similarity matrix between the query and the key, and then multiplies the value by the normalized similarity value. The model is: where d k Indicates the dimension of the key; at the same time, Multi-Head Attention (MHA) projects the query, key, and value into different feature spaces h times, where h represents the number of attention heads, and calculates the attention of each projected query, key, and value in parallel. The final result is obtained by concatenating the results of different heads and projecting them into the final value. The model of MHA is: MHA(Q,K,V)=Concat(head1,…,head h )W O ,in and d v is the dimension of the value and d model is the attention dimension of a single head;

[0023] Preferably, the adaptive fusion weight analysis unit of the adaptive clustering feature fusion module combines the point feature and the cluster feature to which it belongs to calculate a weight w, splices the point feature and the cluster feature to which it belongs together, and sends them to the MLP to obtain the weight w, and uses the sigmoid function to control the weight w within the range of [0,1]. The model of the weight w is: w=Sigmoid(MLP[F,F cluster ]), where F represents the point feature after passing through the backbone network, F cluster It is the clustering feature enhanced by the self-attention mechanism.

[0024] Preferably, the semantic tag prediction module specifically includes the following steps:

[0025] Step 6.1: Use two segmentation heads for both point features and cluster features. There are 4 heads in total. Each head is composed of MLP. The 4 head models are: P semantic =head semantic (F), P mov =head mov (F), P semantic_cluster =head semantic_cluster (F cluster ), P mov_cluster =head mov_cluster (F cluster ), where P semantic and P movThey are the semantic category prediction results and semantic motion prediction results of point features, P semantic_cluster and P mov_cluster They are the semantic category prediction results and semantic motion prediction results of cluster features respectively;

[0026] Step 6.2: Use the weight w obtained in step 5 to perform weighted sum on the prediction results of point features and clusters. The final prediction result model is: P semantic_final =(1-w)·P semantic +w·P semantic_cluster , P mov_final =(1-w)·P mov +w·P mov_cluster ;

[0027] Step 6.3: For the loss function, use cross entropy and lovasz softmax loss functions. The cross entropy loss function model is:

[0028] where y and Represent the true value label and the predicted label respectively, f i Indicates the frequency of category data in the data set; the loss function lovasz softmax model is: Where |C| represents the number of categories, represents the extension of lovasz's jaccard index, x i (c)∈[0,1] and y i (c)∈{-1,1} are the predicted probability and true value label of pixel i for category c, and the final comprehensive loss function model is: L=L wce +L ls .

[0029] Technical effects and advantages of the present invention:

[0030] 1. The present invention encodes the 5-dimensional features of the original 3D points through a linear layer. In order to obtain the local geometric information of the points, the original features of each point and its N neighboring points are first subtracted and sent to a multi-layer perceptron (MLP) for encoding to obtain N such features. Then, the N features are subjected to maximum pooling to obtain the local features of the point. Finally, the local features are fused with the encoded features obtained by the linear layer to obtain the final point features.

[0031] 2. The present invention utilizes the association of laser point cloud time series. Before the current point cloud is sent to the network, the semantic label prior of the current frame is obtained by time series transmission of historical frame prediction labels. The clustering prior information of the foreground object of the current point cloud is obtained by clustering. This information is used to allow the network to adaptively unify the point cloud category prediction results of the foreground object, thereby improving the semantic segmentation accuracy of the network for the foreground object.

[0032] 3. The present invention utilizes time series information to optimize the semantic segmentation results of historical point clouds and current point clouds through a voxel-based voting mechanism, while enhancing the consistency of semantic prediction results in time series, thereby obtaining the final prediction result of the current point cloud to cope with the situation where the time series results of point clouds at different times are inconsistent, which helps to improve the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0034] Figure 2 This is an example diagram of the clustering prior information obtained in the present invention.

[0035] Figure 3 This is a structural diagram of the backbone network algorithm of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] See also Figure 1 As shown, the present invention provides a three-dimensional point cloud semantic segmentation method based on object clustering information, including: a three-dimensional point cloud data acquisition module, a three-dimensional point cloud data processing module, a priori clustering label acquisition module, a point cloud data feature extraction module, an adaptive clustering feature fusion module, a semantic label prediction module, and a voxel semantic label voting module.

[0038] The three-dimensional point cloud data acquisition module is connected to the three-dimensional point cloud data processing module, the prior clustering label acquisition module is respectively connected to the three-dimensional point cloud data processing module and the adaptive clustering feature fusion module, the point cloud data feature extraction module is respectively connected to the three-dimensional point cloud data processing module and the adaptive clustering feature fusion module, and the semantic label prediction module is respectively connected to the point cloud data feature extraction module, the adaptive clustering feature fusion module and the voxel semantic label voting module.

[0039] The three-dimensional point cloud data acquisition module is used to collect the three-dimensional coordinate position information of the vehicle and transmit the position information to the three-dimensional point cloud data processing module;

[0040] The three-dimensional point cloud data processing module is used to process the three-dimensional coordinate position information of the vehicle, obtain the foreground semantic labels of the historical frame point cloud and the points of the multi-frame point cloud voxelization downsampling, and transmit the foreground semantic labels of the historical frame point cloud to the prior clustering label acquisition module, and transmit the points of the multi-frame point cloud voxelization downsampling to the point cloud data feature extraction module;

[0041] The prior clustering label acquisition module is used to cluster the historical frame point cloud foreground semantic labels obtained by the data processing module to obtain the prior clustering labels, and transmit the prior clustering labels to the adaptive clustering feature fusion module;

[0042] The point cloud data feature extraction module is used to extract features from the points of the multi-frame point cloud voxelization downsampling obtained by the data processing module, obtain geometric features under different views, and transmit the geometric features under different views to the adaptive clustering feature fusion module and the semantic label prediction module;

[0043] The adaptive clustering feature fusion module is used to perform clustering feature fusion on the a priori clustering label obtained by the label acquisition module and the point feature obtained by the data feature extraction module to obtain a weight, and transmit the weight to the semantic label prediction module;

[0044] The semantic label prediction module is used to predict the point features obtained by the feature extraction module and the cluster features obtained by the feature fusion module, and transmit the prediction results to the voxel semantic label voting module;

[0045] The voxel semantic label voting module is used to integrate the semantic labels of the historical frame point cloud and the current frame point cloud obtained by the above 6 modules, and obtain the voxel semantic label based on voxel voting, thereby updating the semantic label of the current frame.

[0046] A 3D point cloud semantic segmentation method based on object clustering information includes the following steps: step 1: using a laser radar to collect data on the 3D point cloud position information of a vehicle surface, step 2: processing the laser radar point cloud data collected in step 1, converting the historical frame point cloud to the current frame coordinate system by using position transformation, and downsampling the multi-frame point cloud voxelization, using KNN to obtain the neighboring points of the downsampled points, step 3: transferring the foreground semantic label of the historical frame point cloud obtained by network inference in step 2 to the current frame, clustering it as prior semantic information, and obtaining a priori clustering label, step 4: sending the downsampled points in step 2 and their KNN nearest point information into the network for feature extraction, step 5: performing clustering feature fusion on the priori clustering label obtained in step 3 and the point feature obtained in step 4, step 6: predicting the point feature obtained in step 4 and the clustering feature obtained in step 5, and step 7: voting based on voxels for the semantic label of the current frame point cloud obtained by integrating the historical frame point cloud and the above 6 modules.

[0047] Step 1: Use laser radar to collect data on the three-dimensional point cloud position information of the vehicle surface to provide point cloud information of the object surface for the following steps;

[0048] Step 2: Process the laser radar point cloud data collected in step 1, use position transformation to convert the historical frame point cloud to the current frame coordinate system, and downsample the multi-frame point cloud voxelization, and use KNN to obtain the neighboring points of the downsampled points, which specifically includes the following steps:

[0049] Step 2.1: Perform semantic segmentation on the multi-frame point cloud, align the coordinate systems of the multi-frame point cloud, and use the laser radar's posture transformation to transform the point cloud coordinates at time tk to time t. The transformation model is: in represents the position transformation of the laser radar from time tk to time t, P t-k represents the 3D coordinates of the point cloud in the coordinate system at time tk, P t Represents the 3D coordinates of the point cloud in the coordinate system at time t;

[0050] Step 2.2: In order to make the point cloud data of multiple frames less redundant and more orderly after being spliced ​​together, the point cloud is voxelized, that is, the original point cloud data is converted into a standard voxel with a spatial resolution of W×L×H, where W represents the horizontal width, L represents the horizontal length, and H represents the vertical height. Each voxel only retains one 3D point information, and the information of each 3D point includes intensity information, x, y, z, and distance. At the same time, in order to retain more local geometric information in the feature encoding process, KNN is used to search the nearest N points for the points after voxelization downsampling;

[0051] Step 3: First, the foreground semantic label of the historical frame point cloud obtained by network inference in step 2 is transferred to the current frame as prior semantic information; then, the clustering algorithm is used to cluster the points whose labels in the current frame are foreground categories to obtain the prior clustering labels, which specifically includes the following steps:

[0052] Step 3.1: Remap the historical frame semantic labels into two categories: foreground and background. In the current frame coordinate system, first unlabel the semantic labels of the voxels, then use the labels of the historical points to update the voxel labels containing the historical points. The category of each voxel grid is the category of the multiple points contained in the grid. Finally, the voxel labels are assigned to the points of the current frame, thereby obtaining the prior semantic labels of the current frame.

[0053] See also Figure 2 As shown, step 3.2: use the DBSCAN clustering algorithm to cluster the foreground objects of the current frame. Since the foreground objects in the scene may be dynamic, some foreground objects in the current frame cannot obtain semantic information from the semantic voxels obtained from the historical points, that is, unlabeled. Therefore, in the clustering process, it is necessary to cluster the foreground labeled points and the unlabeled points at the same time; in addition, for relatively sparse points in the distance, it is also easy to obtain unlabel results in the process of transferring historical labels, which makes it easy for the clustering process to obtain clustering noise of such background points. Therefore, after clustering, the clusters that do not contain foreground labeled points are discarded, thereby filtering out the background clustering noise, and finally obtaining the priori clustering results;

[0054] Step 4: Send the downsampled points in step 2 and their KNN nearest point information into the network for feature extraction, including a feature coding analysis unit and a backbone network analysis unit, the feature coding analysis unit is used to encode the original point cloud features and the backbone network analysis unit is used to project the point features to the xy plane, yz plane and xz plane in sequence, and further extract the geometric features under different views, specifically including the following steps:

[0055] Step 4.1: The feature encoding analysis unit performs feature encoding on the original point cloud, and uses the linear layer to encode the 5-dimensional features of the original 3D points. In order to obtain the local geometric information of the point, the original features of each point and its N neighboring points are subtracted and sent to the multi-layer perceptron (MLP) for encoding to obtain N such features. Then, these N features are max-pooled to obtain the local features of the point, and then the local features and the encoded features obtained by the linear layer are fused to obtain the final point features. The encoded feature fusion model is: i =LN([LN(h i ),maxMLP(h j -h i)]), where LN represents the linear layer, h i is the original feature of the current point, h j is the neighboring point p i The original characteristics of

[0056] See also Figure 3 As shown, step 4.2: In the backbone network analysis unit, the point features are first sent to the WI module, that is, according to the voxel coordinates of the point, the point features are projected onto the xy plane, and the point features contained in each pixel in the xy plane are averaged and pooled to obtain the pixel features; secondly, the 2D convolutional network is used to perform convolution operations on the 2D feature map, and the activation function ReLU is selected as the feature activation layer; then the features of the 2D feature map are returned to each point, and the features of the feature map pixels are directly assigned to the 3D points projected to the pixel; finally, the point features are sent to the MLP to enhance the features of the channel dimension. The above process model is: G = F + WI (BN (F)), F = G + MLP (BN (G)), where F ∈ R F×N is the feature of the point, G is the feature of the point after passing through the WI module;

[0057] What needs to be specifically explained in this embodiment is that the point feature is projected onto the yz plane, a 2D convolution operation is performed, and then sent to the MLP, and then the point feature is projected onto the xz plane and sent to the convolution layer and the MLP layer. Such operations of projecting to the xy plane, yz plane and xz plane in sequence are regarded as a stage, and the backbone network includes 16 such stages.

[0058] Step 5: Using the prior clustering labels obtained in step 3, average pooling is performed on the point features obtained in step 4 to obtain clustering features, including a multi-head self-attention mechanism analysis unit and an adaptive fusion weight analysis unit. The multi-head self-attention mechanism analysis unit is used to perform an attention mechanism on the clustering features to strengthen the clustering features, and the adaptive fusion weight analysis unit is used to fuse the clustering features with the point features obtained in step 3 to obtain a weight, specifically including the following steps:

[0059] Step 5.1: In order to make the foreground object have consistent semantic label prediction, the prior clustering information obtained in step 3 is used to average pool the points of the same cluster to obtain the characteristics of the cluster. Due to the different densities of object points, the clustering method may result in the same foreground object being divided into multiple clusters. Therefore, the multi-head self-attention mechanism is used in the analysis unit to enable information interaction between different clusters, thereby strengthening the characteristics of the cluster. The attention mechanism is given by the query matrix Q, the key matrix K and the value matrix V. The attention function calculates the similarity matrix between the query and the key, and then multiplies the value by the normalized similarity value. The model is: where d k Indicates the dimension of the key; at the same time, the attention function usually uses multi-head attention. Multi-Head Attention (MHA) projects the query, key, and value into different feature spaces h times, where h represents the number of attention heads, and calculates the attention of each projected query, key, and value in parallel. The final result is obtained by concatenating the results of different heads and projecting them into the final value. The model of MHA is: MHA(Q,K,V)=Concat(head1,…,head h )W O ,in and d v is the dimension of the value and d model is the attention dimension of a single head;

[0060] Step 5.2: The clustering process may cluster objects that are close to each other into one class. Therefore, the adaptive fusion weight analysis unit combines the point features and the cluster features to which it belongs to calculate a weight w, thereby adaptively fusing the label prediction results of the point and the label prediction results of the cluster. The point features and the cluster features to which they belong are spliced ​​together and sent to the MLP to obtain the weight w. The sigmoid function is used to control the weight w within the range of [0,1]. The model of the weight w is: w = Sigmoid (MLP [F,F cluster ]), where F represents the point feature after passing through the backbone network, F cluster It is the clustering feature enhanced by the self-attention mechanism;

[0061] Step 6: Send the point features of step 4 to two MLPs to predict the semantic category labels and semantic dynamic and static labels of the points respectively. Send the cluster features of step 5 to MLP to predict the semantic category labels and semantic dynamic and static labels of the points belonging to the cluster. Finally, use the weights of step 5 to weight the prediction results of the points and the prediction results of the clusters to obtain the prediction results. Specifically, the following steps are included:

[0062] Step 6.1: Perform semantic segmentation on multi-frame point clouds. Finally, it is necessary to predict the semantic category labels and semantic static and dynamic labels of the point clouds. Therefore, two segmentation heads are used for segmentation of point features and cluster features respectively. There are 4 heads in total, each of which is composed of MLP. The 4 head models are: P semantic =head semantic (F), P mov =head mov (F), P semantic_cluster =head semantic_cluster (F cluster ), P mov_cluster =headmov_cluster (F cluster ), where P semantic and P mov They are the semantic category prediction results and semantic motion prediction results of point features, P semantic_cluster and P mov_cluster They are the semantic category prediction results and semantic motion prediction results of cluster features respectively;

[0063] Step 6.2: Use the weight w obtained in step 5 to perform weighted sum on the prediction results of point features and clusters. The final prediction result model is: P semantic_final =(1-w)·P semantic +w·P semantic_cluster , P mov_final =(1-w)·P mov +w·P mov_cluster ;

[0064] Step 6.3: For the loss function, use cross entropy and lovasz softmax loss functions. The cross entropy loss function model is: where y and Represent the true value label and the predicted label respectively, f i Indicates the frequency of category data in the data set; the loss function lovasz softmax model is: Where |C| represents the number of categories, represents the extension of lovasz's jaccard index, x i (c)∈[0,1] and y i (c)∈{-1,1} are the predicted probability and true value label of pixel i for category c, and the final comprehensive loss function model is: L=L wce +L ls ;

[0065] Step 7: In order to make full use of the temporal information of the laser point cloud and improve the temporal consistency of the semantic prediction results, a voting mechanism is used to integrate the semantic labels independently predicted by the historical frame point cloud and the current frame point cloud to obtain the final prediction result. Therefore, the predicted labels of the 8 frames of historical point cloud and the predicted labels of the current point cloud are used to obtain the semantic voxels in the current frame coordinate system, where the label of each voxel is the label of the largest number of points it contains. The semantic label of the voxel is then returned to the point of the current frame to obtain the final semantic label of the point.

[0066] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0067] The above contents are merely examples and explanations of the concept of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A 3D point cloud semantic segmentation method based on object clustering information, characterized in that: The following steps are involved: Step 1: Collect data on the three-dimensional point cloud position information of the vehicle surface through laser radar; Step 2: Process the point cloud data of the laser radar collected in step 1, convert the historical frame point cloud to the current frame coordinate system using position transformation, and downsample the multi-frame point cloud voxelization, and use KNN to obtain the neighboring points of the downsampled points; Step 3: The foreground semantic label of the historical frame point cloud obtained by network inference in step 2 is transferred to the current frame and clustered as prior semantic information to obtain a priori clustering label; Step 4: Send the downsampled points in step 2 and their KNN nearest point information into the network for feature extraction; Step 5: Perform cluster feature fusion on the prior cluster labels obtained in step 3 and the point features obtained in step 4; Step 6: Predict the point features obtained in step 4 and the cluster features obtained in step 5; Step 7: It is used to integrate the semantic labels of the historical frame point cloud and the current frame point cloud obtained by the above 6 modules, and vote based on voxels.

2. The method for semantic segmentation of three-dimensional point clouds based on object clustering information according to claim 1, characterized in that: The three-dimensional point cloud data processing module specifically includes the following steps: Step 2.1: Perform semantic segmentation on the multi-frame point cloud, align the coordinate systems of the multi-frame point cloud, and use the laser radar's posture transformation to transform the point cloud coordinates at time tk to time t. The transformation model is: in represents the position transformation of the laser radar from time tk to time t, P t-k represents the 3D coordinates of the point cloud in the coordinate system at time tk, P t Represents the 3D coordinates of the point cloud in the coordinate system at time t; Step 2.2: voxelize the point cloud, that is, convert the original point cloud data into standard voxels with a spatial resolution of W×L×H, where W represents the horizontal width, L represents the horizontal length, and H represents the vertical height. Each voxel only retains one 3D point information. The information of each 3D point includes intensity information, x, y, z, and distance. Use KNN to search the nearest N points for the voxelized downsampled points.

3. The method for semantic segmentation of three-dimensional point clouds based on object clustering information according to claim 1, characterized in that: The priori clustering label acquisition module specifically includes the following steps: Step 3.1: Remap the historical frame semantic labels into two categories: foreground and background. In the current frame coordinate system, first unlabel the semantic labels of the voxels, then use the labels of the historical points to update the voxel labels containing the historical points. The category of each voxel grid is the category of the multiple points contained in the grid. Finally, the voxel labels are assigned to the points of the current frame, thereby obtaining the prior semantic labels of the current frame. Step 3.2: Use the DBSCAN clustering algorithm to cluster the foreground objects in the current frame, and cluster the foreground labeled points and unlabeled points at the same time; in addition, after clustering, discard the clusters that do not contain foreground labeled points, thereby filtering out background clustering noise.

4. The method for semantic segmentation of three-dimensional point clouds based on object clustering information according to claim 1, characterized in that: The feature encoding and analysis unit in the point cloud data feature extraction module performs feature encoding on the original point cloud, uses a linear layer to encode the 5-dimensional features of the original 3D points, makes a difference between the original features of each point and its N neighboring points, and sends them to a multi-layer perceptron (MLP) for encoding to obtain N such features, then performs maximum pooling on these N features to obtain the local features of the point, and then fuses the local features with the encoded features obtained by the linear layer to obtain the final point features. The encoded feature fusion model is: i =LN([LN(h i ),maxMLP(h j -h i )]), where LN represents the linear layer, h i is the original feature of the current point, h j is the neighboring point p i original features.

5. The method for semantic segmentation of three-dimensional point clouds based on object clustering information according to claim 1, characterized in that: In the backbone network analysis unit of the point cloud data feature extraction module, the point features are first sent to the WI module, that is, according to the voxel coordinates of the point, the point features are projected onto the xy plane, and the point features contained in each pixel of the xy plane are averaged and pooled to obtain the pixel features; secondly, a 2D convolutional network is used to perform a convolution operation on the 2D feature map, and the activation function ReLU is selected as the feature activation layer; Then the features of the 2D feature map are returned to each point, and the features of the feature map pixels are directly assigned to the 3D points projected to the pixels; finally, the point features are sent to the MLP to enhance the features of the channel dimension. The above process model is: G = F + WI (BN (F)), F = G + MLP (BN (G)), where F ∈ R F×N is the feature of the point, and G is the feature of the point after passing through the WI module.

6. The method for semantic segmentation of three-dimensional point clouds based on object clustering information according to claim 1, characterized in that: The multi-head self-attention mechanism analysis unit of the adaptive clustering feature fusion module uses a multi-head self-attention mechanism, wherein the attention mechanism is given a query matrix Q, a key matrix K and a value matrix V. The attention function calculates the similarity matrix between the query and the key, and then multiplies the value by the normalized similarity value. The model is: where d k Indicates the dimension of the key; at the same time, Multi-Head Attention (MHA) projects the query, key, and value into different feature spaces h times, where h represents the number of attention heads, and calculates the attention of each projected query, key, and value in parallel. The final result is obtained by concatenating the results of different heads and projecting them into the final value. The model of MHA is: MHA(Q,K,V)=Concat(head1,…,head h )W O ,in and d v is the dimension of the value and d model is the attention dimension of a single head.

7. The method for semantic segmentation of three-dimensional point clouds based on object clustering information according to claim 1, characterized in that: The adaptive fusion weight analysis unit of the adaptive clustering feature fusion module combines the point feature and the cluster feature to which it belongs to calculate a weight w, splices the point feature and the cluster feature to which it belongs together, and sends them to the MLP to obtain the weight w, and uses the sigmoid function to control the weight w within the range of [0,1]. The model of the weight w is: w=Sigmoid(MLP[F,F cluster ]), where F represents the point feature after passing through the backbone network, F cluster It is the clustering feature enhanced by the self-attention mechanism.

8. The method for semantic segmentation of three-dimensional point clouds based on object clustering information according to claim 1, characterized in that: The semantic label prediction module specifically includes the following steps: Step 6.1: Use two segmentation heads for both point features and cluster features. There are 4 heads in total. Each head is composed of MLP. The 4 head models are: P semantic =head semantic (F), P mov =head mov (F), P semantic_cluster =head semantic_cluster (F cluster ), P mov_cluster =head mov_cluster (F cluster ), where P semantic and P mov They are the semantic category prediction results and semantic motion prediction results of point features, P semantic_cluster and P mov_cluster They are the semantic category prediction results and semantic motion prediction results of cluster features respectively; Step 6.2: Use the weight w obtained in step 5 to perform weighted sum on the prediction results of point features and clusters. The final prediction result model is: P semantic_final =(1-w)·P semantic +w·P semantic_cluster , P mov_final =(1-w)·P mov +w·P mov_cluster ; Step 6.3: For the loss function, use cross entropy and lovasz softmax loss functions. The cross entropy loss function model is: where y and Represent the true value label and the predicted label respectively, f i Indicates the frequency of category data in the data set; the loss function lovasz softmax model is: Where |C| represents the number of categories, Indicates the extension of lovasz's jaccard index, x i (c)∈[0,1] and y i (c)∈{-1,1} are the predicted probability and true value label of pixel i for category c, and the final comprehensive loss function model is: L=L wce +L ls .

Images