Point cloud sample selection method and apparatus, and computer device

Abstract

The present application relates to the field of point cloud data processing, and more particularly to a point cloud sample selection method and device and computer equipment. The method comprises obtaining a plurality of to-be-labeled point cloud samples; inputting the plurality of to-be-labeled point cloud samples into a pre-trained neural network model for classification detection to obtain the highest and second-highest classification category of each to-be-labeled point cloud sample classification score probability; calculating the minimum margin score and context uncertainty score of the to-be-labeled point cloud sample; and weighting and summing the minimum margin score and context uncertainty score to obtain the uncertainty result of the to-be-labeled point cloud sample; and determining a target labeled point cloud sample according to the uncertainty result of the to-be-labeled point cloud sample.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of point cloud data processing, and particularly relates to a point cloud sample selection method and device and computer equipment. BACKGROUND

[0002] The great success of deep neural networks in the field of computer vision benefits from complex deep network structures and massive labeled data, but in some complex tasks such as three-dimensional point cloud detection and segmentation, the labels of point cloud data are often not easy to obtain, and if all the sample data obtained are labeled, a large amount of human cost is required.

[0003] At present, active learning technology can obtain the most helpful samples for improving the current model from a large amount of unlabeled data, thereby effectively reducing the labeling cost. Lin et al. divide the entire point cloud into segments, and each segment is used as a basic query unit for sample selection. ReDAL suggests selecting sub-scene regions with rich and diverse information for label acquisition. Entropy, color discontinuity and structural complexity are used to measure the information of sub-scene regions. Then, SSDR-AL groups the original point cloud into super points, and gradually selects the most information-rich regions for annotation.

[0004] However, the performance of region-based active learning depends largely on the region division strategy, and once the region division is inaccurate, the selected labeled point cloud cannot meet the accuracy requirements of classification detection. In addition, since point clouds exhibit strong semantic similarity in local regions, selecting all points in a local region will result in a redundant label budget. SUMMARY

[0005] Embodiments of the present application provide a point cloud sample selection method, device and computer equipment to solve the problem of high redundancy and low accuracy of labeled point cloud sample selection based on active learning technology in the current point cloud classification detection task scenario.

[0006] In a first aspect of the present application, a point cloud sample selection method is provided, which comprises:

[0007] obtaining a plurality of unlabeled point cloud samples;

[0008] inputting the plurality of unlabeled point cloud samples into a pre-trained neural network model for classification detection to obtain the class to which each unlabeled point cloud sample belongs with the highest and second highest classification score probability;

[0009] calculating the minimum margin score and the context uncertainty score of the unlabeled point cloud sample; and weighting and summing the minimum margin score and the context uncertainty score to obtain the uncertainty result of the unlabeled point cloud sample;

[0010] According to the uncertainty result of the to-be-labeled point cloud sample, a target labeled point cloud sample is determined.

[0011] In a second aspect of the present application, the present application further provides a point cloud sample selection device, comprising:

[0012] A sample collection module is configured to obtain a plurality of to-be-labeled point cloud samples.

[0013] A sample classification module is configured to input the plurality of to-be-labeled point cloud samples into a pre-trained neural network model for classification detection, so as to obtain the highest and the second highest classification category of each to-be-labeled point cloud sample.

[0014] A sample metric module is configured to calculate the minimum margin score and the context uncertainty score of the to-be-labeled point cloud sample, and to obtain the uncertainty result of the to-be-labeled point cloud sample by weighted summation of the minimum margin score and the context uncertainty score.

[0015] A sample selection module is configured to determine a target labeled point cloud sample according to the uncertainty result of the to-be-labeled point cloud sample.

[0016] In a third aspect of the present application, the present application further provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the point cloud sample selection method according to the first aspect of the present application when executing the computer program.

[0017] The present application has the following advantages:

[0018] The present application groups point clouds in multiple scales, and gradually perceives the context information of unlabeled point clouds in a larger range of levels to calculate the uncertainty score of each point cloud sample, thereby solving the problem that a single point cloud cannot truly reflect the real importance of the point cloud. Using hierarchical minimum margin uncertainty measurement, the context information is fused, which makes it possible to achieve higher performance sampling by annotating point clouds rather than regions, and also solves the problem of region division.

[0019] The present application uses hierarchical minimum margin uncertainty measurement to obtain an output, i.e., a point-level uncertainty score result v u The top k points with the highest uncertainty can be directly selected for labeling. However, this point selection method may concentrate points in local regions as the previous point-based method, resulting in redundant labeling. Therefore, the present application further proposes a feature-distance suppression (FDS) module to ensure uniform and dispersed distribution of points and to ensure that the labeled point information is not redundant.

[0020] This invention provides a hierarchical, reliable, and non-redundant point cloud sample selection method, apparatus, and computer device. This ensures that the selected target point cloud annotation samples are reliable, hierarchical, and non-redundant in information. Furthermore, this invention relies on a region partitioning strategy. Attached Figure Description

[0021] Figure 1 This is a flowchart of the point cloud sample selection method of the present invention;

[0022] Figure 2 This is a structural diagram of the hierarchical minimum marginal uncertainty measurement module of the present invention;

[0023] Figure 3 This is a structural diagram of the feature distance suppression module of the present invention;

[0024] Figure 4 This is a structural diagram of the point cloud sample selection device of the present invention;

[0025] Figure 5 This is a comparison chart of the point cloud sample selection results of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] In some embodiments, the point cloud sample selection method of the present invention can be applied to scenarios where computer devices and servers coexist, wherein the computer devices communicate with the server via a network. The computer devices can be, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server.

[0028] like Figure 1 As shown, a point cloud sample selection method is provided, the method comprising:

[0029] 101. Obtain multiple point cloud samples to be labeled;

[0030] In this embodiment of the invention, the point cloud sample to be labeled can be three-dimensional point cloud data. This three-dimensional point cloud data can be obtained from public datasets or from real-time images captured by devices such as radar. The invention does not limit the source of the three-dimensional point cloud data.

[0031] 102. Input the multiple point cloud samples to be labeled into a pre-trained neural network model for classification detection, so as to obtain the category with the highest and second highest classification score probability for each point cloud sample to be labeled;

[0032] The neural network model is an initial model pre-trained based on a small number of labeled samples. It is used to identify the category to which the point cloud sample to be labeled belongs, so that the importance of the point cloud sample to be labeled can be evaluated based on its category, thereby determining the target labeled point cloud sample.

[0033] 103. Calculate the minimum marginal score and context uncertainty score of the point cloud sample to be labeled; take the weighted sum of the minimum marginal score and context uncertainty score as the uncertainty result of the point cloud sample to be labeled;

[0034] In this embodiment of the invention, considering that relying solely on the point cloud itself without utilizing information from its surrounding environment is insufficient to reflect the true importance of the point cloud, this embodiment designs a Hierarchical Minimum Margin Uncertainty Measurement (HMMUM) module, such as... Figure 2 As shown, by analyzing points at multiple scales The system groups the samples and progressively perceives the contextual information of the point cloud samples to be labeled at a wider level to calculate the uncertainty score v for each point. u Based on the uncertainty score, these candidate point clouds can be identified as target labeled point cloud samples, thus solving the problem that a single point cannot truly reflect its importance. Using hierarchical minimum marginal uncertainty measurement, contextual information is fused, making it possible to achieve higher-performance sampling using only labeled points instead of regions, while also solving the problem of needing to perform region division.

[0035] In some embodiments of the present invention, calculating the minimum marginal fraction of the point cloud sample to be labeled includes:

[0036] The minimum labeling score of the point cloud sample to be labeled is calculated based on the highest and second-highest probabilities of the classification scores.

[0037] Among them, the hierarchical minimum marginal uncertainty measure calculates the uncertainty score for each point by grouping points at multiple scales and progressively perceiving the contextual information of the point cloud samples to be labeled at a wider range of levels. For example... Figure 2 As shown, the point cloud samples containing features to be labeled are input into the hierarchical minimum marginal uncertainty measurement module. We first calculate the minimum marginal fraction U for each point cloud sample containing only its own information. x This can be represented as:

[0038] U x =h(x u ;p1(x u ))-h(x u p2(x) u ))

[0039] Among them, U x Represents the point cloud sample x to be labeled u The minimum marginal fraction of the point cloud sample x to be labeled u This refers to the candidate point cloud that can be labeled; p1(x u p2(x) represents the probability of the highest-scoring label category predicted by the point cloud classification model. u ) represents the probability of the second highest-scoring label category predicted by the point cloud classification model; h(·) represents the segmentation predictor of the point cloud classification model.

[0040] In some embodiments of the present invention, calculating the context uncertainty score of the point cloud sample to be labeled includes:

[0041] Identify the neighboring point cloud samples of the point cloud sample to be labeled at each downsampling layer;

[0042] Obtain the highest and second-highest probabilities of the classification scores of neighboring point cloud samples, and calculate the average highest and second-highest probabilities of the point cloud samples to be labeled in each downsampling layer.

[0043] In this approach, downsampling is used to group a wider range of neighbors. The softmax prediction for each downsampled point is obtained by averaging the softmax predictions of the neighboring point cloud samples in the original unlabeled point cloud sample. The softmax label for each downsampled point represents the predicted distribution of a local region, which we use... This indicates that we performed N layers of voxel downsampling, with the point x to be labeled. u The local context information can be represented by the average prediction of the downsampled i-th layer. The average probability value can be expressed as:

[0044]

[0045] in, Represents the point cloud sample x to be labeled u Local context information in the i-th downsampling layer; Indicates the point cloud sample x to be labeled u The j-th neighbor point cloud sample in the i-th downsampling layer; K iis the total number of neighboring points in the voxel radius in the i-th downsampling layer, and p(·( represents the prediction probability).

[0046] The context uncertainty score of the point cloud sample to be labeled is calculated based on the average highest probability value and the average second highest probability value of each downsampling layer.

[0047] To facilitate the distinction between the average highest probability value and the average second highest probability value, this invention uses subscripts 1 and 2 for differentiation. Therefore, the context uncertainty score of the point cloud sample to be labeled at each downsampling layer is expressed as:

[0048]

[0049] in, Represents the point cloud sample x to be labeled u The context uncertainty score under the i-th downsampling; This indicates the local context information corresponding to the highest score obtained by the point cloud classification model in the i-th downsampling. This represents the local context information corresponding to the second highest downsampling score obtained from the point cloud classification prediction; h(.) represents the classifier of the point cloud classification model.

[0050] In a preferred embodiment of the present invention, after calculating the context uncertainty score of the point cloud sample to be labeled, the method further includes weighted summation of the context uncertainty scores of the point cloud sample to be labeled at each downsampling layer to determine the final context uncertainty score of the point cloud sample to be labeled.

[0051] By weighted summing the minimum marginal score and the contextual uncertainty score, the uncertainty result of the point cloud sample to be labeled can be obtained, expressed as:

[0052]

[0053] Among them, v u Representing point cloud x u Uncertainty of the outcome; w i Let w represent the hyperparameters of the i-th downsampling iteration, and N be the number of downsampling iterations. For example, when N = 3, w i ∈{0.1,0.01,0.001} is a hyperparameter whose values ​​from left to right represent w during the 1st, 2nd, and 3rd downsampling iterations. 1 w 2 w 3 The value of .

[0054] 104. Based on the uncertain results of the point cloud samples to be labeled, determine the target labeled point cloud samples.

[0055] In some embodiments of the present invention, determining the target labeled point cloud sample based on the uncertain results of the point cloud sample to be labeled includes selecting the top-ranked point cloud sample to be labeled as the target labeled point cloud sample based on the ranking of the uncertain results of the point cloud sample to be labeled; or, using a feature distance suppression method, selecting several samples to be labeled with different similarities as the target labeled point cloud sample.

[0056] Using the hierarchical minimum marginal uncertainty measurement, we obtained an output, namely the point-level uncertainty score v. u We can directly select the top k points with the highest uncertainty for labeling. However, considering that this point selection method may have the same problem as traditional point selection methods—since some of the top k points may come from the same local region, it is still possible for points to be concentrated in local regions, which would lead to information redundancy—we designed a feature-distance suppression (FDS) method to ensure that the point distribution is uniform and dispersed. Figure 3 As shown, this ensures that the marker point information is not redundant, and at the same time ensures that the final selected marker points are evenly distributed.

[0057] In a preferred embodiment of the present invention, the feature distance suppression method includes determining a distance suppression radius and a feature similarity threshold; determining whether there are other selected target labeled point cloud samples within the distance suppression radius of the point cloud sample to be labeled; if there are no other selected target labeled point cloud samples, then the point cloud sample to be labeled is used as a target labeled point cloud sample; if there are other target labeled point cloud samples, then the similarity distance between the point cloud sample to be labeled and other target labeled point cloud samples is calculated; if the similarity distance exceeds the feature similarity threshold, then the point cloud sample to be labeled is used as a redundant point and the redundant point is suppressed; if the similarity distance does not exceed the feature similarity threshold, then the point cloud sample to be labeled is used as a target labeled point cloud sample.

[0058] For each selected point x i Given a distance suppression radius r and a feature similarity threshold τ, we first determine whether there are already other selected points within the distance suppression radius of the selected point, D. i It is a set initialized as an empty set, used to store all elements in x. i The point x that has already been selected in the distance suppression radius j Its formula is expressed as:

[0059]

[0060] Where, d ij The representative is x i To xj The distance.

[0061] If, after judgment, D i It is still an empty set, indicating that in x i There are no other selected points in the distance suppression radius, such as Figure 3 As shown in the dashed circle below, the solid black circles represent points that will be selected this time, the solid black squares represent points that have already been selected, and the hollow black circles represent points that have not been selected. So, x... i This will represent this local area, and we will proceed to the next step, which is labeling. Of course, if D... i Not empty Figure 3 The solid circle at the top indicates this. Therefore, we need to determine D. i Points and point x in i A similarity score can be expressed by the formula:

[0062]

[0063] Where f i and f j Representing x respectively i and x j The extracted features, if Sim(x) i ,x j )>τ,x i If a point is considered redundant, it will be removed from the set of points that need to be marked; otherwise, it will be retained for marking.

[0064] Of course, it is understandable that, in addition to using the cosine formula, other similarity formulas can also be used for measurement.

[0065] In one embodiment, a point cloud sample selection device is provided, which corresponds one-to-one with the point cloud sample selection method in the above embodiments. For example... Figure 4 As shown, the point cloud sample selection device 200 includes a sample acquisition module 201, a sample classification module 202, a sample measurement module 203, and a sample selection module 204. Detailed descriptions of each functional module are as follows:

[0066] The sample acquisition module is used to acquire multiple point cloud samples to be labeled.

[0067] The sample classification module is used to input the multiple point cloud samples to be labeled into a pre-trained neural network model for classification detection, so as to obtain the category with the highest and second highest classification score probability for each point cloud sample to be labeled.

[0068] The sample metric module is used to calculate the minimum marginal score and context uncertainty score of the point cloud sample to be labeled; the minimum marginal score and context uncertainty score are weighted and summed to obtain the uncertainty result of the point cloud sample to be labeled.

[0069] The sample selection module is used to determine the target labeled point cloud sample based on the uncertain results of the point cloud sample to be labeled.

[0070] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the sample selection method described in the above embodiments, for example... Figure 1 Steps S101-S104 are shown. Alternatively, the processor executes a computer program to implement the functions of each module / unit in this embodiment of the sample selection device, for example... Figure 4 The functions of each module / unit shown are not described again here to avoid repetition.

[0071] Figure 5 This is a comparison diagram of the point cloud sample selection method of the present invention and the traditional top K point selection strategy. (a) and (c) show the results of selecting the top k uncertainties based on the traditional method, where the points are the selected points. It can be seen that these selected points are not uniform, and are relatively concentrated and redundant in some areas. (b) and (d) show the results of the point cloud sample selection method of the present invention, where the points are the selected points. It can be seen that the points selected by the present invention are more dispersed, less redundant, and have better representativeness.

[0072] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0073] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A point cloud sample selection method, characterized in that, The method includes: Obtain multiple point cloud samples to be labeled; The multiple point cloud samples to be labeled are input into a pre-trained neural network model for classification and detection to obtain the category with the highest and second highest classification score probability for each point cloud sample to be labeled. The process involves calculating the minimum marginal score and context uncertainty score of the point cloud sample to be labeled; then, a weighted sum of the minimum marginal score and context uncertainty score is taken as the uncertainty result of the point cloud sample to be labeled; the calculation of the minimum marginal score and context uncertainty score of the point cloud sample to be labeled includes: calculating the minimum marginal score of the point cloud sample to be labeled based on the highest and second-highest probabilities of the classification scores of the point cloud sample to be labeled; determining the neighboring point cloud samples of the point cloud sample to be labeled in each downsampling layer; obtaining the highest and second-highest probabilities of the classification scores of the neighboring point cloud samples, and calculating the average highest probability value and average second-highest probability value of the point cloud sample to be labeled in each downsampling layer; calculating the context uncertainty score of the point cloud sample to be labeled in each downsampling layer based on the average highest probability value and average second-highest probability value of the point cloud sample to be labeled in each downsampling layer; and finally, a weighted sum of the context uncertainty scores of the point cloud sample to be labeled in each downsampling layer to determine the final context uncertainty score of the point cloud sample to be labeled. Based on the uncertain results of the point cloud samples to be labeled, the target labeled point cloud samples are determined.

2. The point cloud sample selection method according to claim 1, characterized in that, The step of determining the target labeled point cloud sample based on the uncertain results of the point cloud sample to be labeled includes selecting the top-ranked point cloud sample to be labeled as the target labeled point cloud sample based on the ranking of the uncertain results of the point cloud sample to be labeled; or, using the feature distance suppression method, selecting several samples to be labeled with different similarities as the target labeled point cloud sample.

3. The point cloud sample selection method according to claim 2, characterized in that, The feature distance suppression method includes determining the distance suppression radius and the feature similarity threshold; Determine whether there are other selected target labeled point cloud samples within the distance suppression radius of the point cloud sample to be labeled. If there are no other selected target labeled point cloud samples, then the point cloud sample to be labeled is used as a target labeled point cloud sample. If there are other target labeled point cloud samples, calculate the similarity distance between the point cloud sample to be labeled and other target labeled point cloud samples. If the similarity distance exceeds the feature similarity threshold, then the point cloud sample to be labeled is used as a redundant point and the redundant point is suppressed. If the similarity distance does not exceed the feature similarity threshold, then the point cloud sample to be labeled is used as a target labeled point cloud sample.

4. A point cloud sample selection device, characterized in that, include: The sample acquisition module is used to acquire multiple point cloud samples to be labeled. The sample classification module is used to input the multiple point cloud samples to be labeled into a pre-trained neural network model for classification detection, so as to obtain the category with the highest and second highest classification score probability for each point cloud sample to be labeled. The sample metrics module is used to calculate the minimum marginal score and context uncertainty score of the point cloud samples to be labeled; The weighted sum of the minimum marginal score and the contextual uncertainty score is taken as the uncertainty result of the point cloud sample to be labeled; The calculation of the minimum marginal score and context uncertainty score of the point cloud sample to be labeled includes: calculating the minimum marginal score of the point cloud sample to be labeled based on the highest and second-highest probabilities of its classification score; identifying the neighboring point cloud samples of the point cloud sample to be labeled in each downsampling layer; obtaining the highest and second-highest probabilities of the neighboring point cloud samples' classification scores, and calculating the average highest and average second-highest probabilities of the point cloud sample to be labeled in each downsampling layer; calculating the context uncertainty score of the point cloud sample to be labeled in each downsampling layer based on the average highest and average second-highest probabilities of the point cloud sample to be labeled in each downsampling layer; and weighted summing the context uncertainty scores of the point cloud sample to be labeled in each downsampling layer to determine the final context uncertainty score of the point cloud sample to be labeled. The sample selection module is used to determine the target labeled point cloud sample based on the uncertain results of the point cloud sample to be labeled.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the point cloud sample selection method as described in any one of claims 1 to 3.

Images