A semi-supervised point cloud semantic segmentation method based on active learning of point selection

Abstract

The present invention belongs to the field of point cloud data processing, and specifically relates to a semi-supervised point cloud semantic segmentation method based on active learning of point selection; the segmentation method includes obtaining target point cloud data; inputting the target point cloud data into a trained point cloud semantic segmentation neural network, and outputting the segmentation result of the target point cloud data; the labeling cost of the point cloud data is reduced by semi-supervised training of the point cloud semantic segmentation neural network; by actively learning point selection, uncertain points are selected from unlabeled point cloud data sets through a specific sampling strategy, so that the selected point cloud data is important and contains rich information and can also be non-redundant. It can improve the effect of semi-supervised learning and the ability of the model, and ultimately improve the segmentation performance of three-dimensional point cloud data.

Description

Technical Field

[0001] The present invention belongs to the field of point cloud data processing, and in particular relates to a semi-supervised point cloud semantic segmentation method based on active learning of point selection. Background Art

[0002] The goal of semantic segmentation of point clouds is to assign a category label to each 3D point, which can be applied to various scenarios such as robotics, autonomous driving, and augmented reality. Recently, deep learning-based methods have achieved impressive performance. These high-performance methods usually rely on a large amount of data with point-level labels. However, this also brings the problem of high cost and redundant labeling.

[0003] To alleviate the labor and cost of annotation, many methods have been proposed that explore semi-supervised ways to learn 3D segmentation models from a limited number of labeled points, which are usually randomly sampled. These methods attempt to develop effective strategies to propagate label information to unlabeled points. Although these semi-supervised methods can significantly reduce the labeling cost, the performance may be limited. Since these labeled points are randomly selected, the labels may contain redundant information, and important points may be omitted.

[0004] Active learning, as an alternative learning strategy, has recently been studied to alleviate the limitations of 3D segmentation. Lin et al. divided the entire point cloud into segments, and each segment served as a basic query unit for sample selection. ReDAL proposed to select those sub-scene regions that are rich in information and diverse for label acquisition. Entropy, color discontinuity, and structural complexity were used to measure the information of sub-scene regions. Subsequently, SSDR-AL groups the original point cloud into super-points and gradually selects the most informative regions for annotation. However, the performance of region-based active learning depends heavily on the region segmentation strategy. In addition, since point clouds show strong semantic similarity in local regions, selecting all points in a local region will produce a redundant label budget.

[0005] In summary, although the separate use of semi-supervised training and active learning has achieved good results in the point cloud semantic segmentation task, how to effectively combine semi-supervised training and active learning to give full play to the advantages of both is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0006] Based on the problems existing in the prior art, the present invention proposes a semi-supervised point cloud semantic segmentation method based on active learning of point selection, the method comprising acquiring target point cloud data; inputting the target point cloud data into a trained point cloud semantic segmentation neural network, and outputting the segmentation result of the target point cloud data;

[0007] The training process of the point cloud semantic segmentation neural network specifically includes:

[0008] 101. Acquire a point cloud dataset; the point cloud dataset includes a labeled dataset and an unlabeled dataset, the labeled dataset and the unlabeled dataset have the same distribution, each point cloud data in the labeled dataset has a label, and each point cloud data in the unlabeled dataset does not have a label;

[0009] 102. Use labeled and unlabeled datasets to train a point cloud semantic segmentation model in a semi-supervised manner, and predict pseudo labels for unlabeled data;

[0010] 103. A hierarchical minimum margin uncertainty metric is used to calculate the minimum margin score of the point cloud according to the point cloud data labels in the point cloud dataset; and the context uncertainty score of the point cloud is calculated by gradually capturing the local context at different scales based on multiple downsampling;

[0011] 104. Using the minimum margin score and context uncertainty score of the point cloud, the uncertainty result of the point cloud is calculated;

[0012] 105. Based on the uncertainty result of the point cloud, a number of point clouds are selected, and the selected point clouds are labeled and sent to the labeled data set, and are removed from the unlabeled data set, and the process returns to step 101 until the point cloud semantic segmentation model training is completed.

[0013] Beneficial effects of the present invention:

[0014] The present invention realizes the fusion of semi-supervised and improved active learning methods, overcomes the problem that random point selection under the original semi-supervised framework limits the learning performance of the semi-supervised model, and proposes a new hierarchical-feature distance-suppressed point selection active learning point selection strategy, which solves the problems of information redundancy and dependent area division. Through active learning, the present invention realizes high-performance sampling, thereby improving the task performance of point cloud segmentation under the semi-supervised model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of a semi-supervised point cloud semantic segmentation method based on active learning of point selection according to an embodiment of the present invention;

[0016] Figure 2 is a schematic diagram of a point cloud semantic segmentation neural network structure according to an embodiment of the present invention;

[0017] Figure 3 is a schematic diagram of a point cloud semantic segmentation neural network training process according to an embodiment of the present invention;

[0018] Figure 4 It is a schematic diagram of the effect of semi-supervised point cloud semantic segmentation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Figure 1 It is a flow chart of a semi-supervised point cloud semantic segmentation method based on active learning of selected points. Figure 1 As shown, the process of running the segmentation method of the present invention is shown, which specifically includes:

[0021] S1. Obtain target point cloud data;

[0022] In an embodiment of the present invention, the target point cloud data is a set of point cloud data to be segmented, and the category of each point cloud data in the set needs to be marked, and the target point cloud data is segmented according to the category. The point cloud data set to be segmented can be collected by a collection device on an autonomous driving vehicle, for example.

[0023] S2, inputting the target point cloud data into the trained point cloud semantic segmentation neural network, and outputting the segmentation result of the target point cloud data;

[0024] In an embodiment of the present invention, considering that the existing semantic segmentation neural network relies on a large amount of data with point-level labels, which brings about the problems of high cost and redundant labeling, the present invention considers using a semi-supervised point cloud semantic segmentation neural network to reduce the label cost. The semi-supervised point cloud semantic segmentation neural network is pre-trained using deep learning technology. Using the semi-supervised point cloud semantic segmentation neural network, the predicted category and predicted probability of each point cloud data can be predicted, wherein the predicted probability can represent the confidence of the corresponding predicted category, and the segmentation of the target point cloud data can be accurately obtained through the predicted probability and the predicted category.

[0025] In order to better illustrate the semi-supervised point cloud semantic segmentation neural network of the present invention, the present invention describes in detail the semi-supervised point cloud semantic segmentation neural network and its corresponding training process.

[0026] Figure 2 is a schematic diagram of the structure of a point cloud semantic segmentation neural network according to an embodiment of the present invention. Figure 2As shown, the semi-supervised point cloud semantic segmentation neural network adopts a teacher-student model. Specifically, the embodiment of the present invention uses Minkowski space to construct two segmentation networks, which are respectively represented as a teacher model (Teacher Network) and a student model (Student Network). The point cloud data of the point cloud data set is input into the teacher model, and the enhanced data of the point cloud data of the point cloud data set is input into the student model. The consistency constraint between the teacher model and the student model is used, and the teacher model and the student model are trained using cross entropy loss; the student network is optimized by gradient descent; and the parameters of the student model are transferred to the teacher model using an exponential moving average index.

[0027] The formula for transferring the parameters of the student model to the teacher model using the exponential moving average is expressed as:

[0028]

[0029] in, are the parameters of the teacher model during the jth training iteration, are the parameters of the teacher model during the j-1th training iteration, is the parameter of the student model during the jth training iteration; α is a hyperparameter that determines the speed of parameter transfer, which is generally close to 1. In each iteration, the teacher model outputs the predicted labels of the labeled data and the pseudo labels of the unlabeled data. Next, the label output results of this model can be used to sample and select unlabeled points to learn better labeled data through active learning to optimize the point cloud semantic segmentation neural network.

[0030] Figure 3 Schematic diagram of the training process of the point cloud semantic segmentation neural network according to an embodiment of the present invention. Figure 3 As shown, the training process includes:

[0031] 101. Obtain a point cloud data set;

[0032] Considering the cost and redundancy of point cloud data labeling, the present invention selects better point cloud data for labeling through active learning to ensure the accuracy of the point cloud semantic segmentation neural network. The point cloud data set includes a labeled data set and an unlabeled data set. The labeled data set and the unlabeled data set have the same distribution. Each point cloud data in the labeled data set has a label, and each point cloud data in the unlabeled data set does not have a label.

[0033] 102. Use labeled and unlabeled datasets to train a point cloud semantic segmentation model in a semi-supervised manner, and predict pseudo labels for unlabeled data;

[0034] In an embodiment of the present invention, the point cloud data in the labeled data set and the unlabeled data set are respectively input into the teacher model, and the enhanced data of the point cloud data in the labeled data set and the unlabeled data set are input into the student model. The consistency constraint between the teacher model and the student model is used to train the teacher model and the student model using cross entropy loss; the student network is optimized by gradient descent; and the parameters of the student model are transferred to the teacher model using an exponential moving average. In each training process, the teacher model outputs the predicted labels of the labeled data and the pseudo labels of the unlabeled data. Since the labeled data itself has a real label, while the unlabeled data does not have a real label, and the real label of the labeled data is more effective, the real label of the labeled data and the pseudo label of the unlabeled data will be used in the subsequent process to select more important unlabeled data. After manually annotating the selected unlabeled data, the next round of training iteration is entered until the corresponding training requirements are met.

[0035] In some embodiments of the present invention, semi-supervised learning refers to a machine learning method that uses labeled and unlabeled data for training. The entire semi-supervised training process of this embodiment can be reported as:

[0036] 1. Prepare the data set: Prepare the labeled point cloud data and the unlabeled point cloud data separately, and divide them into training set and validation set according to a certain ratio.

[0037] 2. Build a Teacher Network: Use the labeled point cloud data to train a deep neural network as the Teacher network. The training goal of this network is to minimize the loss function of the labeled data, that is, to minimize the gap between the predicted value and the true label value.

[0038] 3. Build a Student network: Use labeled point cloud data and unlabeled point cloud data to train a deep neural network as the Student network. When training the Student network, the output of the Teacher network is used as the label of the unlabeled data, thereby realizing the utilization of the unlabeled data.

[0039] 4. Train the Student network: Iteratively adjust the parameters of the Student network to minimize the loss function of all data, including labeled point cloud data and unlabeled point cloud data. Specifically, in each iteration cycle, the parameters of the Student network are first updated with labeled point cloud data, and then updated with unlabeled point cloud data. The loss function of the unlabeled point cloud data is composed of the gap between the prediction results of the Student network and the output results of the Teacher network.

[0040] 5. Evaluate model performance: After each iteration, use the validation set to evaluate the performance of the Student network. If the performance meets the expected requirements, stop training; otherwise, continue to adjust the parameters of the Student network until the performance requirements are met.

[0041] Through the above steps, the Teacher-Student model can make full use of the unlabeled point cloud data to improve the learning effect, so as to obtain better performance under the same labeled point cloud data, thereby improving the segmentation performance of point cloud semantic segmentation in the subsequent process.

[0042] 103. A hierarchical minimum margin uncertainty metric is used to calculate the minimum margin score of the point cloud according to the point cloud data labels in the point cloud dataset; and the context uncertainty score of the point cloud is calculated by gradually capturing the local context at different scales based on multiple downsampling;

[0043] In the embodiment of the present invention, it is considered that the existing methods of using active learning to perform point cloud segmentation tasks are generally region-based, and the performance of region-based active learning depends to a large extent on the region division strategy. In addition, since point clouds show strong semantic similarity in local areas, selecting all points in local areas will generate a redundant label budget. Therefore, in the active learning adopted by the present invention, the present invention predicts the uncertainty of the point itself and the neighboring points around the point through a hierarchical minimum marginal uncertainty metric, and after multiple downsampling, the local context is gradually captured at different scales. Each downsampling accumulates the information and uncertainty of the neighboring points to obtain a final uncertain result.

[0044] For the hierarchical minimum margin uncertainty metric, specifically, a point without surrounding context information cannot reflect the true importance of the point. The hierarchical minimum margin uncertainty metric calculates the uncertainty score of each point by grouping points at multiple scales and gradually perceiving the context information of unlabeled points at a larger range of levels. The minimum margin score U for each unlabeled point containing only its own information can be calculated as follows x .

[0045] U x =h(x u ; p1(x u ))-h(x u ; p2(x u ))

[0046] Among them, U x Represents point cloud x u The minimum marginal fraction, x u represents the candidate point cloud that can be selected for labeling; p1(x u) represents the probability of the label category with the highest score predicted by the point cloud semantic segmentation model, p2(x u ) represents the probability of the label category with the second highest score predicted by the point cloud semantic segmentation model; h(·) represents the segmentation predictor of the point cloud semantic segmentation model.

[0047] In addition to the minimum labeling score, we need to obtain the point cloud context uncertainty, and we downsample to group a wider range of neighbors. The softmax prediction of each downsampled point is obtained by averaging the softmax predictions of neighboring points in the original point cloud. The softmax label of each downsampled point represents the predicted distribution of a local area, which we use We performed N layers of voxel downsampling, and the point x to be labeled u The local context information of can be represented by the average prediction of the downsampled i-th layer, as shown in the following formula:

[0048]

[0049] in, Represents point x u Local context information at the i-th downsampling layer; Indicates that at the unmarked point x u The jth neighbor point in the i-th downsampling layer; K i is the total number of neighboring points within the voxel radius in the i-th downsampling layer, and p(·) represents the predicted probability. For each downsampling layer, the unlabeled point x u The context uncertainty score It can be obtained by the following formula:

[0050]

[0051] in, Represents point cloud x u Context uncertainty score at the i-th downsampling; It indicates that the local context information corresponding to the highest downsampling score is predicted by the point cloud semantic segmentation model. It indicates that the local context information corresponding to the second highest downsampling score predicted by the point cloud semantic segmentation model is obtained.

[0052] 105. Using the minimum margin score and context uncertainty score of the point cloud, the uncertainty result of the point cloud is calculated;

[0053] In the embodiment of the present invention, by integrating the unlabeled points x u Point-level uncertainty score U xand its contextual uncertainty score The final uncertainty score v is obtained u , expressed as:

[0054]

[0055] Among them, v u Represents point cloud x u The uncertainty result, U x Represents point cloud x u The minimum marginal score of Represents point cloud x u The context uncertainty score at the i-th downsampling, x u represents the candidate point cloud that can be selected for labeling; w i represents the hyperparameter of the i-th downsampling, N is the number of downsampling times. For example, when N = 3, w i ∈{0.1,0.01,0.001} is a hyperparameter whose values ​​from left to right represent the 1st, 2nd, and 3rd downsampling times w 1 , w 2 , w 3 The value of .

[0056] 106. Based on the uncertainty result of the point cloud, a number of point clouds are selected, and the selected point clouds are labeled and sent to the labeled data set, and are removed from the unlabeled data set, and the process returns to step 101 until the point cloud semantic segmentation model training is completed.

[0057] In the embodiment of the present invention, after the hierarchical minimum marginal uncertainty measurement, we obtain a ranking of the final uncertainty. At this time, we can directly select the first K points for marking according to the ranking of the uncertainty results of the point cloud.

[0058] In a preferred embodiment of the present invention, considering that some of the first K points may come from the same local area, which will lead to information redundancy, we have designed a feature distance suppression method to ensure that the final selected marker points are evenly distributed. The feature distance suppression method includes determining the distance suppression radius and the feature similarity threshold; judging whether there are other selected point clouds within the distance suppression radius of the candidate point cloud, if there are no other selected point clouds, the candidate point cloud is used as the selected point cloud; if there are other selected point clouds, the similarity distance between the candidate point cloud and the other selected point clouds, if the similarity distance exceeds the feature similarity threshold, the candidate point cloud is used as a redundant point and the redundant points are suppressed, if the similarity distance does not exceed the feature similarity threshold, the candidate point cloud is used as the selected point cloud.

[0059] Specifically, for each selected point x i, we give a distance suppression radius r and a feature similarity threshold τ. First, we will determine whether there are other selected points within the distance suppression radius of the selected point. i is a set initialized to an empty set, used to store all i The distance suppression radius of the selected point x j , the formula is as follows:

[0060]

[0061] where d ij Represents x i to x j distance.

[0062] If after judgment, D i It is still an empty set, indicating that in x i If there are no other selected points within the distance suppression radius, then x i will represent this local range, and the next step is to mark it. Of course, if D i is not empty, then we need to judge D i The point and point x in i The similarity of is as follows:

[0063]

[0064] where f i and f j Represents x i and x j Extraction features, if Sim(x i ,x j )>τ,x i will be considered as redundant points and will be removed from the set of points that need to be marked.

[0065] Finally, after feature distance suppression, the output is the points we need to mark. Then we can mark these points manually or in other ways. The marked points are marked points. These marked points will be put back into the data set and the marked data set part and the unmarked data set part will be updated. Then the next iteration can be carried out until the preset termination condition is met.

[0066] Figure 4 is a schematic diagram of the effect of semi-supervised point cloud semantic segmentation according to an embodiment of the present invention. Figure 4As shown, the effect diagram of the segmentation results of the present invention compared with the fully supervised method on the test set of S3DIS region-5. When using only 0.43% of labeled data, the method of the present invention has comparable or even better results than the fully supervised baseline (MinkowskiNet).

[0067] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0068] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A semi-supervised point cloud semantic segmentation method based on active learning of selected points, characterized in that: The method comprises acquiring target point cloud data; inputting the target point cloud data into a trained point cloud semantic segmentation neural network, and outputting a segmentation result of the target point cloud data; The training process of the point cloud semantic segmentation neural network specifically includes:

101. Acquire a point cloud dataset; the point cloud dataset includes a labeled dataset and an unlabeled dataset, the labeled dataset and the unlabeled dataset have the same distribution, each point cloud data in the labeled dataset has a label, and each point cloud data in the unlabeled dataset does not have a label; 102. Use labeled and unlabeled datasets to train a point cloud semantic segmentation model in a semi-supervised manner, and predict pseudo labels for unlabeled data; 103. Using the hierarchical minimum margin uncertainty metric, the minimum margin score U of the point cloud is calculated according to the point cloud data labels in the point cloud dataset. x ; And according to multiple downsampling, the local context is gradually captured at different scales, and the context uncertainty score of the point cloud under the i-th downsampling is calculated 104. Using the minimum margin score U of the point cloud x and the context uncertainty score at the i-th downsampling The uncertainty result of the point cloud is calculated and expressed as: Among them, v u Represents point cloud x u The uncertainty result, U x Represents point cloud x u The minimum marginal score of Represents point cloud x u The context uncertainty score at the i-th downsampling, x u represents the candidate point cloud that can be selected for labeling; w i represents the hyperparameter of the i-th downsampling, and N is the number of downsampling times; 105. Based on the uncertainty result of the point cloud, a number of point clouds are selected, and the selected point clouds are labeled and sent to the labeled data set, and are removed from the unlabeled data set, and the process returns to step 101 until the point cloud semantic segmentation model training is completed.

2. According to claim 1, a semi-supervised point cloud semantic segmentation method based on active learning of selected points is characterized in that: The point cloud semantic segmentation neural network includes constructing two point cloud semantic segmentation networks through Minkowski space, which are represented as a teacher model and a student model respectively; inputting the point cloud data of the point cloud data set into the teacher model, inputting the enhanced data of the point cloud data of the point cloud data set into the student model, using the consistency constraint between the teacher model and the student model, and adopting the cross entropy loss to train the teacher model and the student model; optimizing the student network through gradient descent; and transferring the parameters of the student model to the teacher model by using the exponential moving average index.

3. According to claim 2, a semi-supervised point cloud semantic segmentation method based on active learning of selected points is characterized in that: The formula for transferring the parameters of the student model to the teacher model using the exponential moving average is expressed as: in, are the parameters of the teacher model during the jth training iteration, are the parameters of the teacher model during the j-1th training iteration, is the parameter of the student model during the jth training iteration; α is a hyperparameter that determines the speed of parameter transfer.

4. The semi-supervised point cloud semantic segmentation method based on active learning based on point selection according to claim 1, characterized in that: The minimum margin score of the point cloud is expressed as: U x =h(x u ;p1(x u ))-h(x u ;p2(x u )) Among them, p1(x u ) represents the probability of the label category with the highest score predicted by the point cloud semantic segmentation model, p2(x u ) represents the probability of the label category with the second highest score predicted by the point cloud semantic segmentation model; h(·) represents the segmentation predictor of the point cloud semantic segmentation model.

5. The semi-supervised point cloud semantic segmentation method based on active learning based on point selection according to claim 1, characterized in that: The context uncertainty score of the point cloud is expressed as: in, It indicates that the local context information corresponding to the highest downsampling score is predicted by the point cloud semantic segmentation model. It indicates the local context information corresponding to the second highest downsampling score predicted by the point cloud semantic segmentation model; h(·) indicates the segmentation predictor of the point cloud semantic segmentation model.

6. The semi-supervised point cloud semantic segmentation method based on active learning based on point selection according to claim 1, characterized in that: In step 105, the selecting of several point clouds based on the uncertainty results of the point clouds includes selecting several point clouds with high rankings according to the ranking of the uncertainty results of the point clouds; or, selecting several point clouds with different similarities by adopting a feature distance suppression method.

7. The semi-supervised point cloud semantic segmentation method based on active learning based on point selection according to claim 6, characterized in that: The feature distance suppression method includes determining a distance suppression radius and a feature similarity threshold; judging whether there are other selected point clouds within the distance suppression radius of the candidate point cloud, and if there are no other selected point clouds, taking the candidate point cloud as the selected point cloud; if there are other selected point clouds, calculating the similarity distance between the candidate point cloud and the other selected point clouds, and if the similarity distance exceeds the feature similarity threshold, taking the candidate point cloud as a redundant point and suppressing the redundant point, and if the similarity distance does not exceed the feature similarity threshold, taking the candidate point cloud as the selected point cloud.

Images