A 3D point cloud instance segmentation method based on semantic error correction

Abstract

The application discloses a 3D point cloud instance segmentation method based on semantic error correction, and belongs to the technical field of 3D point cloud instance segmentation. The application mainly focuses on reducing the influence of semantic segmentation errors and corrected semantic segmentation on instance segmentation. A difficult semantic segmentation sample correction module is introduced to improve the semantic prediction accuracy of difficult semantic samples. In addition, a set soft clustering module is also proposed, which discards the traditional clustering method of taking semantic prediction as a hard requirement, so that the network has the correction ability to correctly predict instance clustering in the case of semantic prediction error. Through dataset verification, the application is a point cloud instance segmentation method with relatively high efficiency.

Description

Technical Field

[0001] This invention belongs to the field of 3D point cloud instance segmentation technology, specifically relating to a 3D point cloud instance segmentation method based on semantic error correction. Background Technology

[0002] With the development of LiDAR and computer vision technologies, 3D scene understanding of point clouds has attracted increasing attention, especially 3D point cloud instance segmentation. On the one hand, it has wide applications; as a fundamental research area, it can be applied to many fields, such as healthcare, obstacle avoidance, autonomous driving, and augmented reality. On the other hand, its implementation is also quite challenging. Compared to semantic point cloud segmentation, instance segmentation requires not only predicting the semantic label of each point cloud instance but also predicting the label of the instance to which the point cloud belongs.

[0003] Most point cloud instance segmentation methods employ a bottom-up structure. Based on point cloud features, semantic labels and center offset vectors are predicted. Point clouds are then moved near their respective instance center points using the center offset vectors, and point clouds with the same semantic labels are clustered according to their cluster radius to obtain the instance prediction result. Semantic prediction-based point cloud instance segmentation algorithms achieve relatively good instance segmentation results. However, while semantic prediction helps improve the lower bound of instance segmentation, it also limits the upper bound. In fact, the accuracy of semantic segmentation has become one of the most important factors limiting the instance segmentation result. If the semantic label prediction of a point cloud is incorrect, subsequent clustering cannot classify it into the correct cluster, thus failing to obtain the correct instance label.

[0004] Convolutional structures are widely used in 2D image neural networks. However, due to the sparsity and disorder of point clouds, convolutional operations cannot be directly applied to deep neural networks of point clouds. Deep learning methods for point clouds are mainly divided into three categories: multi-view based, point cloud based, and voxel based. Multi-view based methods project point clouds into 2D space for processing, allowing direct processing within traditional 2D networks. Point cloud based methods directly process point clouds, such as PointNet and PointNet++. PointNet uses a multilayer perceptron (MLP) and max pooling to extract global and local features from the point cloud. PointNet++ modularizes PointNet, constructing a hierarchical network through sampling interpolation, improving the ability to extract local features. Voxel-based methods voxelize point clouds, enabling convolutional operations to be used to process point clouds in an ordered space. Most voxel-based methods choose SSCN (Submanifold Sparse Convolutional Network) as their backbone network, constructing a U-Net-style network structure and utilizing sub-stream sparse convolutions to maintain the sparsity of the point cloud and improve computational efficiency.

[0005] Semantic-independent point cloud instance segmentation algorithms typically first generate target proposals and then predict masks for the proposals; Mask-RCNN in the 2D image domain is a typical example. In addition, 3D-SIS combines the color features of RGB-D images with the spatial features of 3D space. 3DBoNet, on the other hand, introduces the concept of bounding boxes from 2D image instance segmentation, proposing 3D bounding boxes for point cloud instance segmentation.

[0006] Semantic-based point cloud instance segmentation algorithms extract features from point clouds using a backbone network, perform semantic prediction on the point clouds in the feature space, and cluster point clouds with the same semantic prediction labels to obtain instance segmentation results. For example, PointGroup combines the clustering results of the original point cloud with the shifted point cloud, and then selects the optimal clustering result through a scoring network. OccuSeg leverages the advantage of the point cloud's own 3D data, using instance occupancy to constrain the merging of point cloud sets. GICN predicts the center point of instances by introducing a Gaussian distribution. HAIS constructs a hierarchical network structure and progressively optimizes clustering layer by layer. SSTNet introduces a semantic supertree, and after constructing the semantic supertree, separates nodes along the construction path, using the final node as the instance segmentation result.

[0007] Semantic prediction transforms the challenging problem of instance segmentation into a relatively easier problem of segmenting instances with the same semantic meaning. Accurate semantic prediction can significantly improve the accuracy of instance segmentation. Due to factors such as incomplete point cloud data due to acquisition issues, low discriminative power between different point cloud categories, and the large distribution range of some point cloud categories making overall feature extraction difficult, semantic segmentation of these point cloud instances is challenging. These samples are defined as semantically difficult point cloud samples. However, most current methods directly select the maximum semantic score as the semantic prediction label. When faced with multiple high semantic scores, insufficient segmentation ability for semantically difficult samples may lead to incorrect semantic predictions. Incorrect semantic predictions significantly reduce the accuracy of instance segmentation. Therefore, a semantic prediction correction module is needed to improve the network's ability to segment semantically difficult samples. Summary of the Invention

[0008] This invention provides a 3D point cloud instance segmentation method based on semantic error correction, which can be used to improve the segmentation accuracy of 3D point cloud instances.

[0009] The technical solution adopted in this invention is as follows:

[0010] A 3D point cloud instance segmentation method based on semantic error correction, the method comprising:

[0011] Step 1: Input the point cloud data containing location and color information into the point cloud instance segmentation network;

[0012] The point cloud instance segmentation network includes a backbone network, an offset prediction branch, a semantic prediction branch, a difficult semantic segmentation sample correction module, a point clustering network, an instance soft clustering network, and an optimization network.

[0013] The backbone network is used to extract point-level features from the input point cloud data, obtain point cloud features, and pass them to the offset prediction branch and the semantic prediction branch respectively. The offset prediction branch is used to predict the offset of the point cloud from the center point of its instance, and the semantic prediction branch is used to predict the semantic score of the point cloud.

[0014] Step 2: Based on the semantic score, identify the difficult samples in the point cloud, use the difficult semantic segmentation correction module to correct the semantic prediction of the difficult samples, and obtain the instance prediction semantic label of each point cloud based on the corrected semantic score, that is, the category with the highest semantic score.

[0015] Step 3: Input the instance prediction semantic label and offset prediction branch output of the point cloud into the point clustering network for point clustering, and obtain the preliminary instance segmentation result based on the point clustering result;

[0016] Step 4: Input the point clustering results output by the point clustering network into the instance soft clustering network to obtain the corrected instance prediction results;

[0017] Step 5: Adjust the corrected instance prediction results by optimizing the network to obtain the final instance segmentation results.

[0018] Preferably, in step 1, the backbone network adopts a 3D U-Net type backbone network. The point cloud data input to the backbone network is first voxelized into multiple voxels. The initial voxel feature of each voxel is the result of average pooling of the point cloud features contained in the voxel. Then, the initial voxel features are input into the U-Net type network for voxel feature extraction. Finally, the extracted voxel features are back-mapped to the point cloud to obtain the point cloud features of each point cloud.

[0019] Preferably, in step 1, the offset of the predicted point cloud of the offset prediction branch from the center point of its respective instance is specifically as follows:

[0020] The method for predicting the offset between a point cloud and the center point of its instance is as follows:

[0021] Predict the center offset vector of the point cloud using a two-layer multilayer perceptron. in, This represents the center offset of the i-th point cloud, where i = 1, 2, ..., N, and N represents the number of point clouds.

[0022] Preferably, in step 1, the semantic prediction branch predicts the semantic scores of each point cloud with respect to various semantic labels using a two-layer multilayer perceptron. i = 1, 2, ..., N, where N represents the number of point clouds.

[0023] Preferably, in step 2, determining the difficult samples in the point cloud based on semantic scores specifically involves:

[0024] Calculate the highest k among the semantic scores of each point cloud. var The variance var between semantic scores i And from all point clouds, the point cloud with the smallest variance is selected as the difficult semantic segmentation sample according to a certain proportion, where k var The specified value.

[0025] Preferably, in step 2, the difficult semantic segmentation correction module includes, in sequence: a splicing layer, a feature extraction network of a U-Net type network, and a two-layer multilayer perceptron;

[0026] The semantic scores of point clouds for difficult samples are reset to 0, and the semantic scores of point clouds for non-difficult samples are reset to 1. Then, the semantic scores of the reset point clouds are concatenated with the point cloud features through a concatenation layer. The concatenation is then input into the feature extraction network of a U-Net-type network for feature extraction. Finally, the extracted features are input into a two-layer multilayer perceptron to obtain the corrected semantic scores.

[0027] Preferably, the maximum value of the semantic score output by the semantic prediction branch is reset to 1, and the rest are reset to 0. Then, the semantic score of the reset point cloud is concatenated with the point cloud features through a concatenation layer, and then input into the feature extraction network of the U-Net type network.

[0028] Preferably, in step 3, the point clustering is performed as follows:

[0029] For each point i, its original coordinates are... The center offset vector obtained from the offset prediction branch Adding them together gives the coordinates of the point cloud after pulling it to the center point of its instances.

[0030] Based on coordinates Point cloud pairs with the same instance predicted semantic label and an Euclidean distance less than a specified value are merged into one class to obtain point cloud clustering (point clustering result).

[0031] Preferably, in step 4, the soft clustering of the set is performed as follows:

[0032] Based on the number of point clouds in the point cluster, the point clusters are divided into primary instance clusters and incomplete instance clusters;

[0033] For each incomplete instance cluster, the normalized semantic score is used as the weight of the Euclidean distance. The Euclidean distance is used to calculate the soft cluster similarity between the incomplete instance cluster and each main instance cluster. The larger the value of the soft cluster similarity, the weaker the similarity between the two.

[0034] The main instance clusters with soft cluster similarity less than or equal to the soft cluster merging threshold are selected to obtain the initial merge set. The main instance clusters with the smallest soft cluster similarity are then found from the initial merge set and used as the current incomplete instance cluster merging objects.

[0035] Preferably, the soft clustering similarity is:

[0036]

[0037] in, Represents the clustering of incomplete instances C i Clustering with primary instance C j The Euclidean distance between the geometric centers of the points is given by the geometric center of the point cluster, which is the coordinates of all points in the cluster. The mean of S(C) i C j ) represents the clustering of incomplete instances C i Clustering of main instances C j The mean semantic score of the point cloud corresponding to the instance predicted semantic label is used, and the main instance clustering is C. j The instance prediction semantic label is determined based on the mean of the semantic scores of point clusters (i.e. the category corresponding to the highest mean), with α and β being two preset parameters.

[0038] Preferably, the soft clustering merging threshold Set to:

[0039]

[0040] Among them, C j P represents the primary instance clustering. Cj This indicates the total number of point clouds included in the primary instance cluster. The primary instance cluster C represents the clustering of the main instances. j The average radius of the category corresponding to the instance predicted semantic label in the training dataset of the point cloud instance segmentation network, where γ represents a preset non-zero parameter.

[0041] Preferably, in step 5, the optimized network sequentially includes: a two-layer SSCN (Submanifold Sparse Convolutional Network) network and a two-layer multilayer perceptron.

[0042] The technical solution provided by this invention brings at least the following beneficial effects:

[0043] The ensemble soft clustering algorithm (instance soft clustering) used in this invention can achieve correct instance clustering even when semantic prediction is incorrect. This invention improves the accuracy of instance segmentation. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram illustrating the processing procedure of a 3D point cloud instance segmentation method based on semantic error correction provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the backbone network structure used in the embodiments of the present invention;

[0047] Figure 3 This is a schematic diagram of the difficult semantic segmentation sample correction module used in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of point clustering in an embodiment of the present invention.

[0049] Figure 5 The diagram below shows a comparison of clustering in an embodiment of the present invention. (5-a) is a diagram of the original set clustering, and (5-b) is a diagram of set clustering using the set soft clustering algorithm provided in an embodiment of the present invention.

[0050] Figure 6 This is a visualization of the experimental results of qualitative experimental analysis in an embodiment of the present invention.

[0051] Figure 7 This is a visualization of the results of a challenging scenario in qualitative experimental analysis, as described in an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0053] This invention provides a 3D point cloud instance segmentation method based on semantic error correction, which aims to solve two problems in the current 3D point cloud instance segmentation: (1) the inconsistency between the feature distribution of difficult samples and the feature distribution of simple samples in 3D point cloud semantic segmentation; (2) the problem that subsequent instance segmentation will definitely be wrong when semantic segmentation is wrong because the point cloud has the same semantic prediction label as a hard prerequisite for instance clustering.

[0054] The present invention addresses the above-mentioned technical problems from two directions: (1) improving the accuracy of semantic prediction, and (2) enhancing the robustness of the network to erroneous semantic predictions.

[0055] The 3D point cloud instance segmentation method based on semantic error correction provided in this embodiment of the invention includes the following steps:

[0056] Step S1: Input the point cloud data with location information (Coords) and color information (Colors) into the point cloud instance segmentation network; wherein, the cloud instance segmentation network includes a backbone network (preferably a 3D U-Net type backbone network), an offset prediction branch, a semantic prediction branch, a hard sample correction module, a point aggregation network, a soft set aggregation network, and a refine network, such as... Figure 1 As shown;

[0057] The backbone network is used to extract point-level features from the input point cloud data, and the resulting point cloud features are then fed into the offset prediction branch and the semantic prediction branch, respectively.

[0058] Among them, the offset prediction branch is used to predict the offset between the point cloud and the center point of its instance, and the semantic prediction branch is used to predict the semantic score of the point cloud.

[0059] Step S2: The semantic prediction of difficult samples is corrected using the difficult semantic segmentation correction module; that is, the input of the difficult semantic segmentation correction module includes the point cloud features output by the backbone network and the prediction results of the semantic prediction branch.

[0060] Step S3: Based on offset prediction and semantic prediction, point clustering is performed to obtain preliminary instance segmentation results; that is, the output of the difficult semantic segmentation correction module and the output of the offset prediction branch are fed into the point clustering network for point clustering.

[0061] Step S4: Based on the instance segmentation results and point cloud semantic prediction, perform instance soft clustering (i.e., set soft clustering) to obtain the corrected instance prediction results. That is, input the point clustering results output by the point clustering network into the instance soft clustering network to obtain the corrected instance prediction results.

[0062] Step S5: Improve (adjust) the corrected instance prediction results by optimizing the network to obtain the final instance segmentation result (Final Instance Prediction).

[0063] As one possible implementation, in step S1, the 3D U-Net-type backbone network for point-level feature extraction is specifically configured as follows:

[0064] like Figure 2 As shown, the backbone network input consists of N point clouds, each containing coordinate information (x, y, z) and color information (r, g, b), for a total of 6 dimensions. The input point clouds are first voxelized into M voxels, and the feature of each voxel is the result of average pooling of the point cloud features contained within that voxel. Then, the voxel features are input into a seven-layer U-Net-type network for further feature extraction. By utilizing downsampling, upsampling, and connection operations within the same layer, voxel features with both global and local information are obtained. Finally, the voxel features are back-mapped back to the point cloud to obtain point cloud features with both global and local information.

[0065] In this embodiment of the invention, the backbone network includes a voxelization module and a U-Net type network. The voxelization module converts the input point cloud into M voxels and extracts the voxel features of each voxel (the features of each voxel are the result of average pooling of the point cloud features contained in the voxel), thereby obtaining an output feature map of dimension M×6, which is then input into the U-Net type network. The U-Net type network includes seven layers, with the left side used for downsampling and the right side used for upsampling. The first layer on the left is a three-layer stacked convolutional layer (conv, stride 1). The second to third layers have the same network structure, consisting of a convolutional layer with a stride of 2 (conv-s2) and two convolutional layers with a stride of 1. Layers 2 to 7 perform downsampling until the output feature map reaches the desired size. The right side includes 6 upsampling layers with the same network structure. Each upsampling layer includes a deconvolutional layer dconv-s2 with a stride of 2, a two-dimensional convolutional layer conv-s2, and a convolutional layer conv. The input of each upsampling layer includes the output of the previous upsampling layer and the output of the same layer on the left. The output of the last upsampling layer is the processing result of M×K dimensional voxel features (K is a preset value, representing the feature dimension). This result is then mapped back to the point cloud to obtain N×K dimensional point cloud features.

[0066] As one possible implementation, in step S1, the offset of the predicted point cloud of the offset prediction branch from the center point of its respective instance is calculated as follows:

[0067] The offset prediction branch predicts the offset between the point cloud and the center point of its respective instance, utilizing a two-layer multilayer perceptron (MLP) and a center offset loss. Based on the acquired point cloud features, the center offset vector of the point cloud is predicted. Among them, center offset loss Used for training the offset prediction branch.

[0068] As one possible implementation, in step S1, the semantic prediction branch predicts the semantic score of the point cloud as follows:

[0069] Using a two-layer multilayer perceptron (MLP) and cross-entropy loss Based on point cloud features, predict the semantic score of the point cloud. Where N is the total number of point clouds, N class The total number of semantic labels. Among them, the cross-entropy loss... Used to train the semantic prediction branch.

[0070] As one possible implementation, in step S2, the difficult semantic segmentation sample correction module specifically includes:

[0071] First, we need to identify difficult semantic segmentation samples from all the samples. This involves setting a metric rule for difficult samples to identify samples that are difficult to segment semantically, which are then designated as difficult samples (difficult semantic segmentation samples).

[0072] The backbone network possesses some semantic segmentation capability, but it is insufficient to correctly segment difficult samples. Intuitively, the semantic scores of the highest-scoring categories for difficult samples do not differ sufficiently, meaning there is insufficient discriminative power to determine the true semantic label of the sample. Therefore, the highest-scoring k categories... var The variance between the (preset) semantic scores should be small. Based on the above analysis, the highest k among the semantic scores of each point cloud is calculated. var The variance var between semantic scores i And from all point clouds, press a var The proportion of (preset value) selects the point cloud with the smallest variance as the sample for difficult semantic segmentation.

[0073] In this embodiment, k var =2, a var =3. To train the network more effectively, this embodiment introduces correct semantic information from simple samples. For example... Figure 3 As shown, the maximum value in the semantic score is set to 1, and the rest are set to 0. This is then concatenated with the features extracted by the backbone network, giving the features more explicit semantic characteristics. Specifically, to improve the learning ability for difficult samples, all values ​​for difficult samples are set to 0 in this embodiment. The aim is to train the network to infer the semantic information of unknown difficult samples using the explicit semantic information around them. Considering efficiency, the extracted point cloud features are input into a network with a similar structure to the backbone network but only three layers for further feature extraction. Finally, a two-layer MLP and cross-entropy loss are used. The corrected semantic scores are obtained. The category corresponding to the highest semantic score in each point cloud is the predicted semantic label for that point cloud. Cross-entropy loss is used in this process. Used for training difficult semantic segmentation sample correction modules.

[0074] As one possible implementation, the point clustering in step S3 is performed as follows:

[0075] For each point i, its original coordinates are... The center offset vector obtained from the offset prediction branch By adding them together, we can obtain the coordinates of the point cloud after pulling it towards the center point of its instances.

[0076] It is worth noting that this embodiment only considers the foreground point cloud, ignoring all background point clouds (such as walls). In the coordinate space after offset prediction, the entire point cloud is traversed, and points with the same semantic prediction label and a Euclidean distance less than r are selected. point Point cloud pairs with a specified value are merged into a single class to obtain a point cloud cluster.

[0077] Figure 4 The midpoint represents the point cloud, and the color of the point indicates the semantic prediction category. Two points with the same color indicate that their semantic prediction labels are the same, and vice versa. The points circled represent point clouds belonging to the same category, that is, point clouds belonging to the same instance.

[0078] As one possible implementation, in step S4, the soft clustering of the set is performed as follows:

[0079] Since the results of clustering are entirely based on semantic prediction, errors in semantic prediction will inevitably lead to errors in the prediction of instances generated by clustering. The ensemble soft clustering mechanism proposed in this invention enables the network to still predict the correct instance labels to a certain extent even when semantic prediction is incorrect.

[0080] First, the point cloud clusters obtained from the point clustering are divided into primary instance clusters and incomplete instance clusters based on the number of points in each cluster. That is, if the number of points in a point cloud cluster reaches a specified number, it is considered a primary instance cluster; otherwise, it is considered an incomplete instance cluster. Due to incorrect semantic prediction or incorrect center offset prediction, incomplete instance clusters are not merged into the correct instance clusters. For each incomplete instance cluster, in this embodiment, the primary instance cluster with the highest soft cluster similarity to it is selected, and the two are merged.

[0081] The soft clustering process of the set is as follows Figure 5 As shown in the figure, the points represent point clouds, and points with the same color indicate that they have the same semantic prediction label. Points within circles belong to the same cluster. Figure 5 In the given example, clusters with only two points are considered incomplete instance clusters, while the rest are considered primary instance clusters. The original ensemble clustering only merges clusters with the same semantic labels, as shown in Figure (5-a). However, the ensemble soft clustering of this invention clusters them based on the semantic scores of the point cloud and the distance between clusters, thereby increasing the probability of obtaining correct instance predictions when the semantic prediction of clustering is incorrect. The processing result is shown in Figure (5-b).

[0082] This invention assumes that the semantic prediction of the primary instance cluster is correct, but the semantic prediction of the incomplete instance cluster may not be correct. Therefore, this invention does not use identical semantic labels as a hard condition for merging clusters. First, the semantic score is passed through a ReLU layer, and then used as a weight to calculate the soft cluster similarity with Euclidean distance. The soft cluster similarity is used as the basis for determining whether clusters can be merged. For soft cluster similarity greater than... Clusters with a soft clustering threshold are considered unlikely to belong to the same cluster and are not merged.

[0083] As one possible implementation, in step S5, the specific processing method for network optimization is as follows:

[0084] Point cloud features belonging to the same predicted cluster are input into a two-layer SSCN network to extract point cloud cluster features. These features are then input into a two-layer MLP to obtain the point cloud mask predictions. Cluster predictions with an IoU (Intersection over Union) greater than 50% of the ground truth (GT) value are considered positive samples, while the rest are considered negative samples. The mask predictions of negative samples contribute very little to the network; therefore, only positive samples are included in the loss calculation. The predicted masks are used to filter background points in the instance predictions, and the MLP is used to predict the confidence level of each instance prediction based on the remaining foreground point features.

[0085] Preferably, the center offset loss The definition is as follows:

[0086]

[0087] Where, p i Represents the i-th point cloud. P is an indicator function. fg The set of points representing the foreground in the entire point cloud. and These represent the predicted center offset vector of point cloud i and the center offset vector of the ground truth, respectively.

[0088] Preferably, the center offset algorithm is as follows:

[0089]

[0090] in, Let i be the original coordinates of point cloud i. Let i be the center offset vector of point cloud i obtained from the offset prediction branch. Let i be the coordinates of point cloud i after it has moved to the vicinity of its instance center point.

[0091] Preferably, the soft clustering similarity algorithm is as follows:

[0092]

[0093] in, It is incomplete instance clustering C i Clustering with primary instance C j The Euclidean distance between the geometric centers, S(C i C j Clustering of incomplete instances C i Clustering of main instances C j The predicted semantic score corresponds to the semantic label, where α and β are hyperparameters, i.e., α and β are preset values. The geometric center of the cluster is the mean of the coordinates of all points in the cluster, and the semantic prediction score of the cluster is the mean of the semantic scores of each category of all points in the cluster.

[0094] Preferably, the preferred value range of α is 1 to 100, and the preferred value range of β is 0.1 to 10. Further, the method of the present invention is optimal when α is 50 and β is 1.

[0095] Preferably, the soft clustering merging threshold Set to:

[0096]

[0097]

[0098] Among them, C j For primary instance clustering, P Cj This represents the total number of point clouds in the primary instance. C j The clustering semantics correspond to the average radius of each category in the dataset. The clustered result after merging is the instance segmentation prediction result after soft clustering. γ is a hyperparameter, with a preferred value of 0.01.

[0099] Preferably, the network loss for training and optimizing the network is the mask prediction loss. The mask prediction loss as follows:

[0100]

[0101] Where, N ins To predict the total number of instances, m i The prediction mask for instance i, Let be the ground truth value of the mask corresponding to instance i, and BCE() be the binary cross-entropy.

[0102] Preferably, the instance confidence loss during training of the entire network. as follows:

[0103]

[0104] Among them, iou i It is the IOU of instance i, score i Let BCE() be the prediction confidence of instance i, and BCE() be the binary cross-entropy.

[0105] The entire network is trained end-to-end using multi-task loss, with an overall loss... The definition is as follows:

[0106]

[0107] in, The semantic cross-entropy loss is used for the semantic prediction branch. The semantic cross-entropy loss of the correction module is applied to difficult semantic segmentation samples. These are the center offset loss, mask prediction loss, and instance confidence loss, respectively.

[0108] In implementing this invention, the first step is to preprocess the acquired point cloud data to ensure that the preprocessed point cloud data matches the input of the network model, such as through operations like normalization and formatting. It should be understood that a point cloud is a massive collection of points representing the spatial distribution and surface characteristics of a target within the same spatial reference frame. Point clouds obtained based on laser measurement principles include three-dimensional coordinates and laser reflection intensity; point clouds obtained based on photogrammetry principles include three-dimensional coordinates and color information; and point clouds obtained by combining laser measurement and photogrammetry principles include three-dimensional coordinates, laser reflection intensity, and color information. After acquiring the spatial coordinates of each sampling point on the object's surface, a collection of points is obtained, referred to as a "point cloud."

[0109] Then, the target point cloud containing location and color information is input into a pre-trained point cloud instance segmentation network (including a backbone network, offset prediction branch, semantic prediction branch, hard semantic segmentation sample correction module, point clustering network, instance soft clustering network, and optimization network) to obtain the instance segmentation result. The point cloud instance segmentation network can directly load pre-trained parameters during training. It should be understood that the point cloud instance segmentation network of this invention is an end-to-end system; inputting the point cloud into the network directly yields the point cloud instance segmentation prediction label result.

[0110] To further verify the processing performance of the method of the present invention, the present invention is verified through the following experimental simulation.

[0111] Experimental Setup: This embodiment selects the ScanNetV2 dataset as the test dataset. The ScanNetV2 dataset is one of the most widely used and powerful datasets in the field of 3D point cloud instance segmentation. It contains 1,613 3D scenes and scans of instances within them, divided into a training set of 1,201 scenes, a validation set of 312 scenes, and a test set of 100 scenes. The instances in the dataset are classified into 18 types. This embodiment trains on the training set and compares with other methods on the test set, and performs ablation experiments on the validation set.

[0112] This embodiment focuses on the segmentation of difficult samples; therefore, it employs relatively stringent AP (Average Precision) and AP. 50 As the primary evaluation metric, AP is provided. 25 The result. AP 25 This indicates that the IoU threshold is 25%, AP 50 The threshold is 50%, while AP represents the average IoU from 50% to 95% with a threshold increment of 5%.

[0113] The model in this embodiment was trained on a single RTX 3090 GPU with a batch size of 4 for 600k iterations. This embodiment sets the voxel size to 0.02m, the initial learning rate to 0.001, and uses the Adam optimizer. Entity soft clustering is not used during training; it is only used as a post-processing step during inference. The clustering radius r... point The soft clustering merging threshold is set to 0.03m. In this context, α is set to 20, β is set to 1, and γ is set to 0.01.

[0114] Quantitative Experimental Analysis: Table 1 shows the comparative experimental results of this embodiment on the ScanNetV2 hidden test set with other state-of-the-art methods. This embodiment achieves AP and AP... 50 and AP 25The accuracy rates for the three indicators were 46.2%, 70.0%, and 78.1%, respectively.

[0115] Table 1

[0116]

[0117] Qualitative experimental analysis: This embodiment visualizes the experimental results on the ScannetV2 validation set, such as... Figure 6 As shown, the excellent performance of this method in point cloud instance segmentation can be clearly seen. To illustrate the experimental results of soft clustering, this embodiment visualizes some challenging scenarios on the ScannetV2 validation set, such as... Figure 7 As shown. In these scenarios, the network failed to correctly predict the semantic labels of some point clouds. This embodiment compares the visualization effects of using soft clustering and not using soft clustering. It can be found that with the help of soft clustering, our network has the ability to correctly predict instance labels even when semantic prediction is wrong.

[0118] Ablation Experiment Analysis: In this embodiment, an ablation experiment was conducted to investigate the impact of the difficult semantic segmentation sample correction module and the set soft clustering module on the final instance segmentation. The experimental results are shown in Table 2.

[0119] Table 2

[0120]

[0121]

[0122] This embodiment conducted an ablation experiment on the impact of the difficult semantic segmentation sample correction module on the semantic segmentation effect. The experimental results are shown in Table 3.

[0123] Table 3

[0124]

[0125] This embodiment conducted an ablation experiment to investigate the impact of different preprocessing methods on semantic scores before soft clustering of sets. The experimental results are shown in Table 4.

[0126] Table 4

[0127]

[0128] This embodiment conducted an ablation experiment to investigate the influence of different parameters in the soft clustering module. The experimental results are shown in Table 5.

[0129] Table 5

[0130]

[0131] Runtime Analysis: Practical applications of point cloud instance segmentation often have high real-time requirements, such as autonomous driving. This embodiment also compares the running efficiency of SSEC and other point cloud instance segmentation algorithms on the ScanNetV2 validation set, as shown in Table 6.

[0132] Table 6

[0133]

[0134] As shown in Table 6, this embodiment takes as little as 211 milliseconds per scene, demonstrating very high computational efficiency.

[0135] This invention proposes a novel point cloud instance segmentation algorithm, primarily focusing on reducing the impact of semantic segmentation errors and correcting these errors on instance segmentation. A difficult semantic segmentation sample correction module is introduced to improve the semantic prediction accuracy for difficult semantic samples. Furthermore, an ensemble soft clustering module is proposed, abandoning traditional clustering methods that treat semantic prediction as a rigid requirement, enabling the network to correctly predict instance clusters even when semantic prediction errors occur. The effectiveness of the algorithm is verified on the ScanNetV2 dataset, a commonly used and authoritative dataset in the field of point cloud deep learning. This invention also demonstrates a highly efficient point cloud instance segmentation algorithm.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0137] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A 3D point cloud instance segmentation method based on semantic error correction, characterized in that, Includes the following steps: Step 1: Input the point cloud data containing location and color information into the point cloud instance segmentation network; The point cloud instance segmentation network includes a backbone network, an offset prediction branch, a semantic prediction branch, a difficult semantic segmentation sample correction module, a point clustering network, an instance soft clustering network, and an optimization network. The backbone network is used to extract point-level features from the input point cloud data, obtain point cloud features, and pass them to the offset prediction branch and the semantic prediction branch respectively. The offset prediction branch is used to predict the offset of the point cloud from the center point of its instance, and the semantic prediction branch is used to predict the semantic score of the point cloud. Step 2: Based on the semantic score, identify the difficult samples in the point cloud, use the difficult semantic segmentation correction module to correct the semantic prediction of the difficult samples, and obtain the instance prediction semantic label of each point cloud based on the corrected semantic score. Step 3: Input the instance prediction semantic label and offset prediction branch output of the point cloud into the point clustering network for point clustering, and obtain the preliminary instance segmentation result based on the point clustering result; Step 4: Input the point clustering results output by the point clustering network into the instance soft clustering network for ensemble soft clustering to obtain the corrected instance prediction results; Step 5: Adjust the corrected instance prediction results by optimizing the network to obtain the final instance segmentation results; The difficult semantic segmentation correction module includes, in sequence, a concatenation layer, a feature extraction network of a U-Net-type network, and a two-layer multilayer perceptron. The semantic score of the point cloud of difficult samples is reset to 0, and the semantic score of the point cloud of non-difficult samples is reset to 1. Then, the concatenation layer concatenates the semantic score of the point cloud with the point cloud features. After that, the feature extraction network of the U-Net-type network is input for feature extraction. The extracted features are then input into the two-layer multilayer perceptron to obtain the corrected semantic score. In step 4, the soft clustering of the set is performed as follows: Based on the number of point clouds in the point cluster, the point clusters are divided into primary instance clusters and incomplete instance clusters; For each incomplete instance cluster, the normalized semantic score is used as the weight of the Euclidean distance. The Euclidean distance is used to calculate the soft cluster similarity between the incomplete instance cluster and each main instance cluster. The larger the value of the soft cluster similarity, the weaker the similarity between the two. The main instance clusters with soft cluster similarity less than or equal to the soft cluster merging threshold are selected to obtain the initial merge set. The main instance clusters with the minimum soft cluster similarity are then found from the initial merge set and used as the current incomplete instance cluster merging objects. The soft clustering similarity is: in, Representing incomplete instance clustering Clustering with primary instances The Euclidean distance between the geometric centers of the points is given by the geometric center of the point cluster, which is the coordinates of all points in the cluster. The mean, Representing incomplete instance clustering Clustering of main instances The mean semantic score of the point cloud corresponding to the instance predicted semantic label, and the main instance clustering. The instance prediction semantic labels are determined based on the mean of the semantic scores of point clusters. and These are two preset parameters.

2. The method as described in claim 1, characterized in that, In step 1, the backbone network adopts a 3D U-Net type backbone network. The point cloud data input to the backbone network is first voxelized into multiple voxels. The initial voxel feature of each voxel is the result of average pooling of the point cloud features contained in the voxel. Then, the initial voxel features are input into the U-Net type network for voxel feature extraction. Finally, the extracted voxel features are back-mapped to the point cloud to obtain the point cloud features of each point cloud.

3. The method as described in claim 1, characterized in that, In step 1, the offset between the predicted point cloud of the offset prediction branch and the center point of its respective instance is as follows: Predict the center offset vector of the point cloud using a two-layer multilayer perceptron. ,in, This represents the center offset of the i-th point cloud, where i = 1, 2, ..., N, and N represents the number of point clouds.

4. The method as described in claim 1, characterized in that, In step 1, the semantic prediction branch uses a two-layer multilayer perceptron to predict the semantic scores of each point cloud for various semantic labels. , i=1,2,…,N, where N represents the number of point clouds.

5. The method as described in claim 1, characterized in that, In step 2, the determination of difficult samples in the point cloud based on semantic scores specifically involves: Calculate the highest semantic score among all point clouds. Variance between semantic scores And from all point clouds, the point cloud with the smallest variance is selected as the difficult semantic segmentation sample according to a certain proportion, where, The specified value.

6. The method as described in claim 1, characterized in that, In step 3, the point clustering method is as follows: For each point its original coordinates The center offset vector obtained from the offset prediction branch Adding them together gives the coordinates of the point cloud after pulling it to the center point of its instances. ; Based on coordinates Point cloud pairs with the same instance predicted semantic label and an Euclidean distance less than a specified value are merged into one class to obtain the point clustering result.

7. The method as described in claim 1, characterized in that, soft clustering merging threshold Set to: in, Indicates the primary instance clustering, This indicates the total number of point clouds included in the primary instance cluster. Represents the primary instance clustering The average radius of the category corresponding to the instance predicted semantic label in the training dataset of the point cloud instance segmentation network. This indicates a preset non-zero parameter. and These are two preset parameters.

Images